We're trying to use WebClient in our microservice to get large data object from external backend service, and the microservice runs on Kubernetes cluster (I tested on both Docker Desktop Kubernetes or Azure Kubernetes). We're seeing the pod memory go up until the pod is killed and restarted.

I filed an issue for Reactor Netty. We found that this problem does not occur when vanila Reactor Netty and Spring Boot 2.7.6 and above is used. This problem remains when WebClient and Spring Boot 2.7.2 and above is used.

Expected Behavior

The pod should not be killed, and the memory usage should be similar or less than using the RestTemplate.

Actual Behavior

When we make external service calls using Netty's WebClient, we're seeing the memory increases seemingly without limit. I used JCMD to check GC and get the heapdump. GC seems normal and there is no leak found from the heapdump. If I run the microservice as a standalone app, I confirmed the GC happened with VisualVM.

When I run the microservices on Docker Desktop Kubernetes, the memory is consistently going up until the instance is eventually killed without any outOfMemory errors.

Steps to Reproduce

I've created sample projects for all the things I tried. The code is in test-service-client. It has 3 branches.

The backend: test-backend branch. For testing purpose, I simplify the result that has about 8mb, 16mb, 24mb and 32mb bogus data. The actual external service returns a big object that contains different information. You can run "kubectl apply -f test-backend.yaml" to create the pod. The Tomcat WebClient: test-webclient-tomcat branch. The microservice runs with -Xmx250M. You can run "kubectl apply -f test-webclient-tomcat.yaml" to create the pod. The Tomcat RestTemplate: test-rest-template branch. The microservice runs with -Xmx250M. You can run "kubectl apply -f test-rest-template.yaml" to create the pod.

After you checkout the branch, you can run "gradle docker" to build the docker image and deploy the docker image to any kubernetes cluster. The pod memory resource is set to 320M. You can monitor the memory usage using Kubernetes Dashboard or "kubectl describe podMetrics ".

After the pods are deployed and port forward to a local port, I trigger the Tomcat WebClient service one at a time. The pod will be killed or restarted after 4 or 6 requests are completed.

@Service
public class DataServiceImpl implements DataService {

  private static final Logger log = LoggerFactory.getLogger(DataServiceImpl.class);

  @Value("${app.source.url}")
  private String sourceUrl;

  @Autowired private WebClient webClient;

  public DataResult getData(String imageNo) {
    try {
      var httpHeaders = new HashMap<String, String>();
      var responseStr =
          webClient
              .get()
              .uri(sourceUrl + (StringUtils.hasText(imageNo) ? "/" + imageNo : ""))
              .headers(headers -> headers.setAll(httpHeaders))
              .retrieve()
              .bodyToMono(String.class)
              .block();

      if (StringUtils.hasText(responseStr)) {
        var result = new DataResult();
        result.setCode(DataResultCode.OK);
        result.setData(responseStr);
        return result;
      }
    } catch (Exception e) {
      log.error("Failed...", e);
      var result = new DataResult();
      result.setCode(DataResultCode.ERROR);
      result.setMessage(e.getMessage());
      return result;
    }
    return null;
  }
}

Your Environment

  • Spring Boot version(s) used: 2.7.2 - 2.7.7
  • Reactor version(s) used: Spring Webflux 5.3.22, Reactor Core 3.4.23
  • Reactor Netty version: 1.0.23
  • JVM version (java -version): OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)
  • OS and version (eg. uname -a): Docker image adoptopenjdk/openjdk11

Comment From: rstoyanchev

For testing purpose, I simplify the result that has about 8mb, 16mb, 24mb and 32mb bogus data.

The logic in the backend controller looks a little different. It seems to start with 24 MB and doubles on every subsequent request, except on every 4th.

The pod will be killed or restarted after 4 or 6 requests are completed.

Could you make it clear how the data size grows in the request sequence?

Comment From: lopenn01

Could you make it clear how the data size grows in the request sequence?

The instance variable, data, holds approximately 8MB, and the variable lastOrder stores the last order number. When the method is accessed, lastOrder increments by 1. Then dividing lastOrder by 4 returns the remainder. The method generates the returned data using data and remainder.

I added some debug statements to the controller. Here is the log. 2023-01-17 09:32:32.861 DEBUG 37976 --- [nio-9350-exec-2] com.example.demo.controller.Controller : order remainder: [0], data byte length=[8493912] 2023-01-17 09:32:42.503 DEBUG 37976 --- [nio-9350-exec-6] com.example.demo.controller.Controller : order remainder: [1], data byte length=[16987824] 2023-01-17 09:32:49.033 DEBUG 37976 --- [nio-9350-exec-1] com.example.demo.controller.Controller : order remainder: [2], data byte length=[25481736] 2023-01-17 09:32:56.232 DEBUG 37976 --- [nio-9350-exec-4] com.example.demo.controller.Controller : order remainder: [3], data byte length=[33975648] 2023-01-17 09:33:05.193 DEBUG 37976 --- [nio-9350-exec-5] com.example.demo.controller.Controller : order remainder: [0], data byte length=[8493912]

Comment From: rstoyanchev

Thanks for clarifying and also for preparing a sample. That said it would save a lot of time if you make the sample match exactly the test scenario, and remove or factor out code for alternative test scenarios.

As explained in https://github.com/reactor/reactor-netty/issues/2590#issuecomment-1369575966, Netty uses direct memory for faster network I/O, but it's also more expensive to create, and so it is pooled, which requires extra memory. I'm guessing 4-6 requests is with direct, pooled memory? That is not a level comparison with the RestTemplate. Can you clarify what numbers you are seeing with unpooled, heap memory, and likewise the numbers you get with the RestTemplate?

Comment From: lopenn01

Thanks for clarifying and also for preparing a sample. That said it would save a lot of time if you make the sample match exactly the test scenario, and remove or factor out code for alternative test scenarios.

My sample code is very simple, and I don't see any test code. Can you clarify which logic causes the problem?

I have the same problem using WebClient with unpooled memory. Please see my comment in reactor/reactor-netty#2590 (comment).

Comment From: patrik-huber

Hello, just wanted to reach out to you to ask if there is currently work happening on that bug. We are facing this problem as well and we built a workaround by removing the WebClient completly for bigger requests. We really want to use the WebClient again to get rid of our workaround. Thank you for your help already in advance.

Comment From: PatrikHubster

Wanted to check in one more time. We really depend on a fix of this bug. Do you have any updates on it?

Comment From: snicoll

@patrik-huber I am afraid we don't. I am not sure I get the link with Kubernetes though, if something is not releasing memory as it should then we should be able to reproduce that on a plain JVM as well.

If you can rework the sample so that it is easier for us to reproduce the issue, I am sure it'll have a positive effect on the outcome.

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.