Expected behavior Want to get a solution for troubleshooting and resolving this exception
Actual behavior Jedis throws an exception during program operation：TooManyClusterRedirectionsException: Too many Cluster redirections?
Steps to reproduce: Please create a reproducible case of your problem. Make sure that case repeats consistently and it's not random 1.Connect aws redis using java and jedis 2.During the program operation, modify the AWS Redis node type, which lasts about 20 minutes. During the modification process, AWS Redis will add available nodes and kill the previous nodes. At this time, the IP of these nodes will change. I use the automatic reconnection function of jedis to enable the program to automatically reconnect to the cluster endpoint without restarting 3.Redis can be operated normally before, during and after the modification, until the program continues to run for several hours, and I receive Too many Cluster Redirections exceptions. I will put the complete stack information below
Redis / Jedis Configuration Jedis version: spring-boot-stater-data-redis 2.1.10 jedis 2.9.3
Redis version: Amazon MemoryDB for redis db.r6g.large
Java version:
Comment From: stary24
except. org.springframework.data.redis.TooManyClusterRedirectionsException: Too many Cluster redirections?; nested exception is redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections? at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:61) at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41) at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) at org.springframework.data.redis.connection.jedis.JedisClusterConnection.convertJedisAccessException(JedisClusterConnection.java:760) at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.convertJedisAccessException(JedisClusterStringCommands.java:556) at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.setEx(JedisClusterStringCommands.java:207) at org.springframework.data.redis.connection.DefaultedRedisConnection.setEx(DefaultedRedisConnection.java:295) at org.springframework.data.redis.core.DefaultValueOperations$4.potentiallyUsePsetEx(DefaultValueOperations.java:268) at org.springframework.data.redis.core.DefaultValueOperations$4.doInRedis(DefaultValueOperations.java:261) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:225) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:185) at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96) at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:256) at team.smb.ucs.util.redis.RedisUtil.set(RedisUtil.java:184) at team.smb.ucs.api.ezgateway.exception.GlobalExceptionHandle.handleException(GlobalExceptionHandle.java:271) at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:409) at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:61) at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:141) at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80) at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1299) at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1111) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1057) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at brave.servlet.TracingFilter.doFilter(TracingFilter.java:65) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:94) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at brave.servlet.TracingFilter.doFilter(TracingFilter.java:82) at org.springframework.cloud.sleuth.instrument.web.TraceWebServletAutoConfiguration$LazyTracingFilter.doFilter(TraceWebServletAutoConfiguration.java:143) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at team.smb.ucs.api.ezgateway.filter.AuthFilter.doFilter(AuthFilter.java:151) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at team.smb.ucs.util.filter.RequestIdFilter.doFilter(RequestIdFilter.java:75) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at team.smb.ucs.util.filter.ModifyHttpServletFilter.doFilter(ModifyHttpServletFilter.java:65) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1787) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:750) Caused by: redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections? at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:95) at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:135) at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:135) at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:135) at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:135) at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:135) at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:58) at redis.clients.jedis.BinaryJedisCluster.setex(BinaryJedisCluster.java:283)
Comment From: stary24
redis: database: 0 cluster: nodes: test-redis.xxxxxxxxxxx.amazonaws.com:6379 jedis: pool: max-total: 10 max-idle: 10 min-idle: 2 test-on-borrow: true max-wait-millis: -1 block-when-exhausted: true
Comment From: stary24
The reason for using jedis is that I need its automatic reconnection function, which is not available in Lettuce in my Redis version
Comment From: zuiderkwast
AWS specific problem?
Comment From: madolson
"AWS specific"ish. The problem is how we do modifications of clusters. In our service, we vend DNS endpoints for TLS clusters that we modify the IP addresses for. We have seen issues where if you aren't re-resolving the DNS, you might be talking to the old node serving the slot, getting a DNS endpoint that points to the same node, and you retry until it gives up. Some people have also had a lot of success reducing the JVM DNS TTL cache by updating using these instructions (https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html#how-to-set-the-jvm-ttl).
One other solution is to switch to IP based endpoints in TLS, (which is unfortunately only available in Redis 7 which is not supported by MemoryDB). We are supposed to have a fixed for that rolled out, but it seems to keep getting delayed. Since we want to keep this project free of managed provider specific problems, please follow up with AWS support if you have further questions. (Going to close this)
Comment From: stary24
"AWS specific"ish. The problem is how we do modifications of clusters. In our service, we vend DNS endpoints for TLS clusters that we modify the IP addresses for. We have seen issues where if you aren't re-resolving the DNS, you might be talking to the old node serving the slot, getting a DNS endpoint that points to the same node, and you retry until it gives up. Some people have also had a lot of success reducing the JVM DNS TTL cache by updating using these instructions (https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html#how-to-set-the-jvm-ttl).
One other solution is to switch to IP based endpoints in TLS, (which is unfortunately only available in Redis 7 which is not supported by MemoryDB). We are supposed to have a fixed for that rolled out, but it seems to keep getting delayed. Since we want to keep this project free of managed provider specific problems, please follow up with AWS support if you have further questions. (Going to close this)
Why can my program continue to be used when the IP address of the cluster has just changed, until it suddenly becomes unavailable after two hours? I think jedis has automatically reconnected the new IP for me at the beginning
Comment From: stary24
AWS specific problem?
I'm not sure. I only use AWS Redis cluster