Getting following connection error with AWS Elasticache (production)
2024-10-19T08:33:26.374Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : SAVING AUTHORIZATION REQUEST
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : Saved Authorization Request org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@117c1dd in WebSession: org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2LoginSpec$OidcSessionRegistryWebFilter$OidcSessionRegistryServerWebExchange$OidcSessionRegistryWebSession@1420e939
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : Authorization Request state: O2Ltdm7yaOIpkBYzAl7k5YY0XktgoAqh3T04kYfIfYI=
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] c.f.b.a.r.OAuth2ServerRedirectStrategy : RUNNING SERVER REDIRECT STRATEGY: 302 FOUND
2024-10-19T08:33:29.682Z INFO 1 --- [BFFApplication] [or-http-epoll-2] .b.a.r.t.CustomServerCsrfTokenRepository : Loading CSRF token
2024-10-19T08:33:29.684Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.h.c.SPACsrfTokenRequestHandler : Handling CSRF token: MonoSwitchIfEmpty
2024-10-19T08:33:29.684Z INFO 1 --- [BFFApplication] [or-http-epoll-2] .b.a.r.t.CustomServerCsrfTokenRepository : Generating CSRF token
2024-10-19T08:33:29.685Z INFO 1 --- [BFFApplication] [or-http-epoll-2] f.b.a.r.s.RedisSecurityContextRepository : REMOVING AUTHORIZATION REQUEST
2024-10-19T08:33:29.692Z INFO 1 --- [BFFApplication] [xecutorLoop-3-1] f.b.a.r.s.RedisSecurityContextRepository : Removed authorization request
2024-10-19T08:33:29.692Z INFO 1 --- [BFFApplication] [xecutorLoop-3-1] f.b.a.r.s.RedisSecurityContextRepository : Successfully deserialized Authorization Request: org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@705f23d3
io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
2024-10-19T08:33:29.695Z WARN 1 --- [BFFApplication] [or-http-epoll-1] r.netty.http.client.HttpClientConnect : [04ff363b-2, L:/10.0.141.193:43530 - R:dev-xxxxxxxxxx.auth0.com/104.18.35.70:443] **The connection observed an error, the request cannot be retried as the headers/body were sent**
Caused by: io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
The only message I see is:
The connection observed an error, the request cannot be retried as the headers/body were sent
If it does pass that stage:
I later also get this 2nd error:
2024-10-19T09:13:07.654Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.m.GrantedAuthoritiesMapperConfig : PROCESSING AUTHORITY: OIDC_USER
2024-10-19T09:13:07.654Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.m.GrantedAuthoritiesMapperConfig : OidcUserAuthority ID Token Claims
2024-10-19T09:13:07.660Z INFO 1 --- [BFFApplication] [xecutorLoop-3-5] c.f.b.a.s.CustomSessionIdGenerator : Successfully generated session ID: BFF-1729329187658-c73aad09-7fd2-4980-9fc5-0c2704e97ec4
org.springframework.data.redis.RedisSystemException: Error in execution
Caused by: io.lettuce.core.RedisCommandExecutionException: CROSSSLOT Keys in request don't hash to the same slot
The only message I see here is:
CROSSSLOT Keys in request don't hash to the same slot
Local Laptop Development runs fine (using Redis.com cache)
2024-10-19T09:49:51.937+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : SAVING AUTHORIZATION REQUEST
2024-10-19T09:49:51.938+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] c.f.b.a.r.OAuth2ServerRedirectStrategy : RUNNING SERVER REDIRECT STRATEGY: 302 FOUND
2024-10-19T09:49:51.938+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : Authorization Request state: 81PGq8kSVgHdOxfqxj_awoELdV5qvFwu50xKuyHSICI=
2024-10-19T09:49:51.938+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : Saved Authorization Request org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@14e4bf33 in WebSession: org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2LoginSpec$OidcSessionRegistryWebFilter$OidcSessionRegistryServerWebExchange$OidcSessionRegistryWebSession@6fa54c91
2024-10-19T09:49:55.569+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] .b.a.r.t.CustomServerCsrfTokenRepository : Loading CSRF token
2024-10-19T09:49:55.569+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] .b.a.r.t.CustomServerCsrfTokenRepository : Generating CSRF token
2024-10-19T09:49:55.570+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] c.f.b.a.h.c.SPACsrfTokenRequestHandler : Handling CSRF token: MonoSwitchIfEmpty
2024-10-19T09:49:55.574+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-2] f.b.a.r.s.RedisSecurityContextRepository : REMOVING AUTHORIZATION REQUEST
2024-10-19T09:49:55.610+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-8] f.b.a.r.s.RedisSecurityContextRepository : Successfully deserialized Authorization Request: org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@40a82415
2024-10-19T09:49:55.610+01:00 INFO 96570 --- [BFFApplication] [xecutorLoop-3-8] f.b.a.r.s.RedisSecurityContextRepository : Removed authorization request
2024-10-19T09:49:56.746+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-4] c.f.b.a.m.GrantedAuthoritiesMapperConfig : PROCESSING AUTHORITY: OIDC_USER
2024-10-19T09:49:56.747+01:00 INFO 96570 --- [BFFApplication] [ctor-http-nio-4] c.f.b.a.m.GrantedAuthoritiesMapperConfig : OidcUserAuthority ID Token Claims
There is an old issue on Github that talks about the 1st error:
https://github.com/reactor/reactor-netty/issues/1774
The 2nd error is due to some Redis Cluster config issue, but I can't find any Spring documentation about it.
Here is my Redis Connection factory config, but I can't see where I am going wrong
@Configuration
internal class RedisConnectionFactoryConfig(
private val springDataProperties: SpringDataProperties,
private val profileProperties: ProfileProperties
) {
// reactive RedisConnectionFactory for key expiration event handling
@Bean
@Primary
fun reactiveRedisConnectionFactory(): ReactiveRedisConnectionFactory {
// configure Redis standalone configuration
val config = RedisStandaloneConfiguration()
config.hostName = springDataProperties.redis.host
config.port = springDataProperties.redis.port
config.setPassword(RedisPassword.of(springDataProperties.redis.password))
// create client options
// Create SSL options if SSL is required
val sslOptions = SslOptions.builder()
.jdkSslProvider() // Or use OpenSslProvider if you prefer
.build()
// Create timeout options
val timeoutOptions = TimeoutOptions.builder()
.fixedTimeout(Duration.ofSeconds(20))
.timeoutCommands(true)
.build()
// cluster specific settings for optimal reliability.
val clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
.enablePeriodicRefresh(Duration.ofSeconds(5))
.dynamicRefreshSources(false)
.adaptiveRefreshTriggersTimeout(Duration.ofSeconds(5))
.enableAllAdaptiveRefreshTriggers().build()
// create socket options
val socketOptions = SocketOptions.builder()
.keepAlive(SocketOptions.DEFAULT_SO_KEEPALIVE)
.tcpNoDelay(SocketOptions.DEFAULT_SO_NO_DELAY)
// time to wait for connection to be established, before considering it as a failed connection
.connectTimeout(Duration.ofSeconds(60))
.build()
val mappingFunction: (HostAndPort) -> HostAndPort = { hostAndPort ->
val host = springDataProperties.redis.host
val addresses: Array<InetAddress> = try {
DnsResolvers.JVM_DEFAULT.resolve(host)
} catch (e: UnknownHostException) {
e.printStackTrace()
emptyArray() // Handle error and return an empty array
}
val cacheIP = addresses.firstOrNull()?.hostAddress ?: ""
var finalAddress = hostAndPort
if (hostAndPort.hostText == cacheIP) {
finalAddress = HostAndPort.of(host, hostAndPort.port)
}
finalAddress
}
val resolver = MappingSocketAddressResolver.create(DnsResolvers.JVM_DEFAULT, mappingFunction)
// customize thread pool size
val clientResources = DefaultClientResources.builder()
.ioThreadPoolSize(8)
.computationThreadPoolSize(8)
.socketAddressResolver(resolver)
.build()
val clusterClientOptionsBuilder = ClusterClientOptions.builder()
.autoReconnect(true)
.pingBeforeActivateConnection(true)
.timeoutOptions(timeoutOptions)
.socketOptions(socketOptions)
.topologyRefreshOptions(clusterTopologyRefreshOptions)
.validateClusterNodeMembership(true)
.suspendReconnectOnProtocolFailure(true)
.disconnectedBehavior(DEFAULT_DISCONNECTED_BEHAVIOR)
.decodeBufferPolicy(DecodeBufferPolicies.ratio(0.5F))
.requestQueueSize(1000)
.maxRedirects(DEFAULT_MAX_REDIRECTS)
.publishOnScheduler(true) //DEFAULT_PUBLISH_ON_SCHEDULER.
.protocolVersion(ProtocolVersion.RESP3) // Use RESP3 Protocol to ensure AUTH command is used for handshake.
// conditionally use sslOptions if profileProperties.active is 'prod'
if (profileProperties.active == "prod") {
clusterClientOptionsBuilder.sslOptions(sslOptions)
}
// build the clientClusterOptions configuration
val clusterClientOptions = clusterClientOptionsBuilder.build()
// configure connection pool settings
fun buildLettucePoolConfig(): GenericObjectPoolConfig<Any> {
val poolConfig = GenericObjectPoolConfig<Any>()
poolConfig.maxTotal = 100
poolConfig.maxIdle = 50
poolConfig.minIdle = 10
poolConfig.setMaxWait(Duration.ofSeconds(120))
poolConfig.timeBetweenEvictionRuns = Duration.ofSeconds(120)
poolConfig.minEvictableIdleTime = Duration.ofMinutes(5)
poolConfig.testOnBorrow = true
poolConfig.testWhileIdle = true
poolConfig.testOnReturn = true
poolConfig.blockWhenExhausted = true
poolConfig.lifo = true
return poolConfig
}
// create Lettuce client configuration with authentication details
val clientConfigBuilder = LettucePoolingClientConfiguration.builder()
// maximum time allowed for a Redis command to execute before the operation is considered timed out.
.commandTimeout(Duration.ofSeconds(60))
.clientResources(clientResources)
.clientOptions(clusterClientOptions)
.poolConfig(buildLettucePoolConfig())
// conditionally enable SSL only if profileProperties.active is 'prod'
if (profileProperties.active == "prod") {
clientConfigBuilder.useSsl()
}
// build the clientConfig configuration
val clientConfig = clientConfigBuilder.build()
// create Lettuce connection factory
return LettuceConnectionFactory(config, clientConfig).apply {
afterPropertiesSet()
validateConnection = false
setShareNativeConnection(true)
}
}
}
Can someone help?
Comment From: sjohnr
@dreamstar-enterprises thanks for reaching out. Unfortunately, the issue you're reporting here does not look to be related to Spring Security in any way since we do not have any Redis integration in this library. I'm going to close this but please provide more information if you feel I have misunderstood.
Comment From: dreamstar-enterprises
Hi Steve, apologies. I found the right Spring Team group. I posted it here: https://github.com/spring-projects/spring-data-redis/issues/3027
Separately, I am stuck on a problem with Spring. I was thinking of storing my Spring BFF in an AWS Autoscaling group - but with multiple BFFs, it would mean having multiple circuit breakers (with potentially different circuit breaker states, depending on where the AWS Load Balancer sends requests too). I couldn't work out how to solve this (unless I only had one BFF instance). Would I need to reach out to the Spring Cloud Gateway Team for help on this?
Comment From: sjohnr
Thanks for clarifying, @dreamstar-enterprises.
Separately, I am stuck on a problem with Spring.
That sounds like a question best suited for stack overflow but I agree that circuit breakers would likely be related to Spring Cloud.