Hello team,
We are observing an issue that is reproducible 100%.
With this code:
@Bean
public WebClient getWebClient(final WebClient.Builder builder, final SslContext sslContext) {
final var clientHttpConnector = new ReactorClientHttpConnector(HttpClient.create().wiretap(true).metrics(true, Function.identity()).protocol(HttpProtocol.HTTP11).secure(sslContextSpec -> sslContextSpec.sslContext(sslContext).handshakeTimeout(Duration.ofMinutes(1L)).closeNotifyFlushTimeout(Duration.ofMinutes(1L)).closeNotifyReadTimeout(Duration.ofMinutes(1L))));
return builder.baseUrl(hostAndPort).defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE).clientConnector(clientHttpConnector).build();
}
This issue will occur 100% of the time:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.reactive.function.client.WebClient$Builder]: Factory method 'webClientBuilder' threw exception with message: Error creating bean with name 'webClientHttpConnectorCustomizer' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webClientHttpConnectorCustomizer' parameter 0: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: Cannot reserve 49152 bytes of direct buffer memory (allocated: 10452747, limit: 10485760)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:647)
... 63 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webClientHttpConnectorCustomizer' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webClientHttpConnectorCustomizer' parameter 0: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: Cannot reserve 49152 bytes of direct buffer memory (allocated: 10452747, limit: 10485760)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:542)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1164)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:561)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1696)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1653)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1470)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1392)
at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.resolveStream(DefaultListableBeanFactory.java:2198)
at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.orderedStream(DefaultListableBeanFactory.java:2192)
at org.springframework.boot.autoconfigure.web.reactive.function.client.WebClientAutoConfiguration.webClientBuilder(WebClientAutoConfiguration.java:55)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140)
... 64 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: Cannot reserve 49152 bytes of direct buffer memory (allocated: 10452747, limit: 10485760)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:639)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1164)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:561)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:907)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:785)
... 84 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: Cannot reserve 49152 bytes of direct buffer memory (allocated: 10452747, limit: 10485760)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:647)
... 98 common frames omitted
Caused by: java.lang.OutOfMemoryError: Cannot reserve 49152 bytes of direct buffer memory (allocated: 10452747, limit: 10485760)
at java.base/java.nio.Bits.reserveMemory(Unknown Source)
at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)
at io.netty.channel.unix.Buffer.allocateDirectWithNativeOrder(Buffer.java:40)
at io.netty.channel.epoll.EpollEventArray.<init>(EpollEventArray.java:58)
at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:101)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:190)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:37)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:49)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:117)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:81)
at reactor.netty.resources.DefaultLoopEpoll.newEventLoopGroup(DefaultLoopEpoll.java:93)
at reactor.netty.resources.DefaultLoopResources.cacheNativeServerLoops(DefaultLoopResources.java:262)
at reactor.netty.resources.DefaultLoopResources.cacheNativeClientLoops(DefaultLoopResources.java:226)
at reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:145)
at reactor.netty.tcp.TcpResources.onClient(TcpResources.java:263)
at reactor.netty.transport.NameResolverProvider.newNameResolverGroup(NameResolverProvider.java:535)
at reactor.netty.transport.ClientTransportConfig.lambda$getOrCreateResolver$0(ClientTransportConfig.java:249)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
at reactor.netty.internal.util.MapUtils.computeIfAbsent(MapUtils.java:63)
at reactor.netty.transport.ClientTransportConfig.getOrCreateResolver(ClientTransportConfig.java:248)
at reactor.netty.transport.ClientTransport.runOn(ClientTransport.java:352)
at reactor.netty.transport.ClientTransport.runOn(ClientTransport.java:42)
at reactor.netty.transport.Transport.runOn(Transport.java:249)
at org.springframework.http.client.reactive.ReactorClientHttpConnector.lambda$applyLoopResources$1(ReactorClientHttpConnector.java:117)
at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
at org.springframework.http.client.reactive.ReactorClientHttpConnector.createHttpClient(ReactorClientHttpConnector.java:110)
at org.springframework.http.client.reactive.ReactorClientHttpConnector.<init>(ReactorClientHttpConnector.java:101)
at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory.createClientHttpConnector(ReactorClientHttpConnectorFactory.java:68)
at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory.createClientHttpConnector(ReactorClientHttpConnectorFactory.java:44)
at org.springframework.boot.autoconfigure.web.reactive.function.client.ClientHttpConnectorFactory.createClientHttpConnector(ClientHttpConnectorFactory.java:32)
at org.springframework.boot.autoconfigure.web.reactive.function.client.ClientHttpConnectorAutoConfiguration.webClientHttpConnector(ClientHttpConnectorAutoConfiguration.java:57)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140)
... 99 common frames omitted
What is strange is that if we replace the code with:
@Bean
public WebClient getWebClient(final SslContext sslContext) {
final var clientHttpConnector = new ReactorClientHttpConnector(HttpClient.create().wiretap(true).metrics(true, Function.identity()).protocol(HttpProtocol.HTTP11).secure(sslContextSpec -> sslContextSpec.sslContext(sslContext).handshakeTimeout(Duration.ofMinutes(1L)).closeNotifyFlushTimeout(Duration.ofMinutes(1L)).closeNotifyReadTimeout(Duration.ofMinutes(1L))));
return WebClient.builder().baseUrl(hostAndPort).defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE).clientConnector(clientHttpConnector).build();
}
The issue is fixed.
This leads us to think this WebClient.Builder builder is causing the memory issue.
Therefore, we would like to reach out with this issue ticket.
Thank you for your time reading this.
Comment From: bclozel
It does not for me, I can run this code successfully. Can you share a sample application that reproduces this?
The following message suggests that the JVM is not allowed to reserve more direct memory. Maybe the JVM is in a constrained environment or some JVM memory flags restrict this behavior? In this case this wouldn't be a Spring Boot issue but rather a problem with the application setup.
java.lang.OutOfMemoryError: Cannot reserve 49152 bytes of direct buffer memory