Expected Behavior When my session store is down (I'm using spring session with Redis but it should be applicable to any store), if the filter chain is not requiring a session (SessionCreationPolicy.NEVER, IF_REQUIRED and STATELESS) and I'm passing a session cookie (because my browser does so), I expect my service to work normally since the session cookie is not required.

Current Behavior Currently, the SessionManagementFilter, if a session cookie is present, will validate that the requested session id is valid whether a session is actually required or not. In the spring session redis implementation, this will trigger a query to Redis and will throw a runtime exception if Redis is down for a maintenance or because of an issue.

Context This can be reproduced by having a service with a session cookie set on /. 2 filter chains : /api/ -> stateless / -> stateful (login page, authorization endpoint, etc.)

If you visit /login, this will inject a session cookie. Now, if you take down your session store and visit /api in your browser, it will trigger an error.

If you clear the cookie and refresh, it will not trigger an error anymore since the condition request.getRequestedSessionId() != null is false is SessionManagementFilter therefore not triggering the problematic code path.

I'm trying to be more resilient to maintenances and outages happening on my session store.

Thanks!

Comment From: rwinch

I'm not sure I follow your scenario, so perhaps an example would help. However, from what I see this is necessary so that SessionManagementFilter can send a user to the session expiration page. I'm not sure how else it would do that other than checking if the session id is valid. Keep in mind that SessionManagementFilter checks the session every request (not just one that require authorization) because users may be leveraging session for something else and expect the filter to work then too.

Comment From: jebeaudet

I've put up a small project here explaining what I'm trying to achieve. The only requirement is a redis running locally on the default 6379 port.

Project description

SessionRequiredController A small controller that requires a session. The path /session/ping will store a UUID in the session and return its value all the time. This is analogous to a AuthenticationSuccessHandler destination that would mint a bearer token based on the session's authentication.

StatelessController A small stateless controller. The path /stateless/ping will return pong no matter what. This is analogous to a real stateless api with authentication based on a bearer token.

WebSecurityConfiguration The application defines 2 security chains : 1. One on /session/** with the lowest precedence and with a SessionCreationPolicy.ALWAYS session creation policy; 2. One on /stateless/** with the highest precedence and a SessionCreationPolicy.STATELESS session creation policy.

How to reproduce my issue

  1. Boot the service with redis running
  2. Assert that /session/ping and /stateless/ping return the correct values.
  3. You should have a JSESSIONID with the / path which mean that this cookie will get sent on all the requests of our example
  4. shut down redis
  5. Try to access /session/ping -> You get a 500 which is expected because a session is required
  6. Try to access stateless/ping -> You get the same 500 page which is what I'm trying to avoid
  7. Clear your JSESSIONID cookie in your browser
  8. Try to access /session/ping -> You get the whitelabel error page with a 500
  9. Try to access stateless/ping -> it works!

Discussion

Just to reiterate my enhancement request, I'd like to be able to make point 6 work, even if you have a session cookie in your browser. I'm fine with the exceptions of point 5 and 8, they are expected. I'm aware that there might be implications to what I ask for but I really think it would be a nice addition.

Workaround

I've actually coded a workaround but I'm not too fond of it, basically I wrap the HttpServletRequest right before the SessionRepositoryFilter and override those method : 

    @Override
    public HttpSession getSession(boolean create)
    {
        HttpSession httpSession = null;
        //we want the exception thrown when a session is required and redis is down
        if (create) {
            httpSession = super.getSession(create);
        } else {
            try {
                httpSession = super.getSession(create);
            } catch (RedisConnectionFailureException e) {
                logger.debug("Swallowed redis connection exception since session creation is not required!", e);
            }
        }

        return httpSession;
    }

    @Override
    public boolean isRequestedSessionIdValid()
    {
        try {
            return super.isRequestedSessionIdValid();
        } catch (RedisConnectionFailureException e) {
            String requestUri = getRequestURI();
            if (requestUri.startsWith("/stateless")) {
                logger.debug("Swallowed redis connection exception for request URI '{}' since session creation is not required!",
                             requestUri,
                             e);
                return true;
            }
            throw e;
        }
    }

Thanks for your time!

Comment From: rwinch

Thanks for the update. I will try and take a look at it in the next few days.

Comment From: jebeaudet

@rwinch Any chance you had the time to look at this? Thanks

Comment From: rwinch

Thanks for the ping @jebeaudet. I'm not sure if this is the proper place to solve the problem as there are lots of places accessing the HttpSession.

Remember that SessionCreationPolicy.STATELESS is only for how Spring Security authenticates. It cannot control every aspect of HttpSession since the HttpSession can be accessed outside of Spring Security.

I've updated Spring Security's configuration to no longer access the session for your example. However, Spring Framework's FrameworkServlet still tries to obtain the session. The stacktrace looks like this:

2020-05-07 13:59:50.352 ERROR 10623 --- [nio-8080-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host localhost:6379
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:281) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:475) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:134) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:97) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:84) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:215) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.DefaultHashOperations.entries(DefaultHashOperations.java:245) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.data.redis.core.DefaultBoundHashOperations.entries(DefaultBoundHashOperations.java:183) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    at org.springframework.session.data.redis.RedisIndexedSessionRepository.getSession(RedisIndexedSessionRepository.java:440) ~[spring-session-data-redis-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.session.data.redis.RedisIndexedSessionRepository.findById(RedisIndexedSessionRepository.java:412) ~[spring-session-data-redis-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.session.data.redis.RedisIndexedSessionRepository.findById(RedisIndexedSessionRepository.java:249) ~[spring-session-data-redis-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getRequestedSession(SessionRepositoryFilter.java:351) ~[spring-session-core-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getSession(SessionRepositoryFilter.java:289) ~[spring-session-core-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.getSession(SessionRepositoryFilter.java:192) ~[spring-session-core-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:244) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:244) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:244) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.web.util.WebUtils.getSessionId(WebUtils.java:359) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.publishRequestHandledEvent(FrameworkServlet.java:1145) ~[spring-webmvc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1023) ~[spring-webmvc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) ~[spring-session-core-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) ~[spring-session-core-2.2.2.RELEASE.jar:2.2.2.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.5.RELEASE.jar:5.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
    at java.base/java.lang.Thread.run(Thread.java:835) ~[na:na]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host localhost:6379
    at redis.clients.jedis.Connection.connect(Connection.java:204) ~[jedis-3.1.0.jar:na]
    at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:100) ~[jedis-3.1.0.jar:na]
    at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1866) ~[jedis-3.1.0.jar:na]
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:276) ~[spring-data-redis-2.2.6.RELEASE.jar:2.2.6.RELEASE]
    ... 89 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na]
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[na:na]
    at java.base/java.net.Socket.connect(Socket.java:591) ~[na:na]
    at redis.clients.jedis.Connection.connect(Connection.java:181) ~[jedis-3.1.0.jar:na]
    ... 92 common frames omitted

At this point, I don't think there is anything Spring Security can do from here. Please let me know your thoughts.

Comment From: jebeaudet

You're right that since HttpSession could be accessed through the code outside of Spring Security, it's hard to make a polished solution for this. With my custom SessionRepositoryFilter it would work since WebUtils#getSessionId uses HttpServletRequest#getSession(false) and my custom code handles that case but it would publish an event with a null session ID which is not be accurate.

I guess WebUtils could use the HttpServletRequest#getRequestedSessionId method but that doesn't guarantee that the ID returned is valid of even the one used for the session (we can easily think of session fixation on authentication success).

My code doesn't rely on HttpSession outside of the security layer so I'm fine with keeping my custom SessionRepositoryFilter and it works well but I would have rather use something built in and something that others could benefit too.

I'll close this for now, if you have any other ideas I'd be thrilled to hear them but I'm running out of options here.

Once again, thanks for your help!