We have 40 VMs (8 core, 27 GB RAM) running with Spring Webflux (version given below). Randomly facing performance issue at 2-5 VMs i.e. normal response time is 3-10 ms and in the problematic VMs response time is ~1000 ms.

I have analysed the API flow with Dynatrace and found that performance issue at MediaType.parseMediaType(), more than 75% of the response time contribution is from MediaType.parseMediaType.

Spring WebFlux Version

io.projectreactor reactor-bom Dysprosium-SR4 pom import

Embedded Server:

org.springframework.boot spring-boot-starter-jetty ${spring-boot.version}

Comment From: radhakrishnanch

Dynatrace snapshot

Comment From: radhakrishnanch

Thread Dump

thread_dump.txt

Comment From: radhakrishnanch

Heap Dump -

I could see java.util.concurrent.ConcurrentLinkedQueue inside MimeTypeUtils.cachedMimeTypes consumes 37.12mb. This queue contains 1,621,030 items. I assume this is the root cause of the slowness.

All items inside the queue are null

Heap Dump is uploaded at Google Drive https://drive.google.com/file/d/1tTqw2brE6RaNb8LLtOzLxM7H6bPkXFt_/view?usp=sharing

Comment From: bclozel

Which Spring Framework version are you using?

Comment From: bclozel

This looks like a duplicate of #24886

Comment From: radhakrishnanch

Which Spring Framework version are you using?

Spring Boot 2.2.4.RELEASE

Comment From: radhakrishnanch

24886

I could see we have a StringUtils.hasLength(mimeType) check before adding the value to the cache. Still how the null values are added to the queue?

Comment From: bclozel

Spring Boot 2.2.4 depends on Spring Framework 5.2.3. I think this issue is a duplicate of #24886 and I’ll close it as one.

Please upgrade to Spring Boot 2.2.7 or override the Spring Framework version to 5.2.6 in the Spring Boot dependency management.

Thanks!

Comment From: xbj110825

24886

I could see we have a StringUtils.hasLength(mimeType) check before adding the value to the cache. Still how the null values are added to the queue?

jdk(1.8.0_60) ConcurrentLinkedQueue.remove method has bug: https://bugs.openjdk.org/browse/JDK-8137185

You can use the following code to test whether ConcurrentLinkedQueue has bug

public class CLQBug
{
    public static void main(String[] args)
    {
        ConcurrentLinkedQueue<String> queue = new ConcurrentLinkedQueue<>();
        queue.add("x");

        Method first = getNonPublicFirstMethod();
        Method succ = getNonPublicSuccMethod();

        long start = System.currentTimeMillis();
        long end;

        for (long iterations = 0; iterations < Long.MAX_VALUE; iterations++) {
            queue.add("" + iterations);
            queue.remove("" + iterations);

            if (iterations % 10000 == 0) {
                end = System.currentTimeMillis();
                System.out.println("actualSize = " + actualSize(queue, first, succ) + ", elapsed time = " + (end - start));
                start = end;
            }
        }
    }

    public static long actualSize(ConcurrentLinkedQueue<String> queue, Method first, Method succ) {
        long b = 0;
        try {
            for (Object node = first.invoke(queue); node != null; node = succ.invoke(queue, node)) {
                ++b;
            }
        } catch (IllegalAccessException | InvocationTargetException e) {
            throw new RuntimeException(e);
        }
        return b;
    }

    public static Method getNonPublicFirstMethod() {
        Method first;
        try {
            first = ConcurrentLinkedQueue.class.getDeclaredMethod("first");
        } catch (NoSuchMethodException e) {
            throw new RuntimeException(e);
        }
        first.setAccessible(true);
        return first;
    }

    public static Method getNonPublicSuccMethod() {
        Method succ = null;
        for (Method declaredMethod : ConcurrentLinkedQueue.class.getDeclaredMethods()) {
            if (declaredMethod.getName().equals("succ")) {
                succ = declaredMethod;
                break;
            }
        }
        if (succ == null) {
            throw new RuntimeException("not found");
        }
        succ.setAccessible(true);
        return succ;
    }
}

if ConcurrentLinkedQueue has bug

[root@0bcfb0e0e13e work]# /home/work/1.8.0_60/bin/java -jar main.jar
elapsed time = 0
elapsed time = 331
elapsed time = 654
elapsed time = 1085
elapsed time = 1536

else

[root@0bcfb0e0e13e work]# /usr/bin/java -jar main.jar
elapsed time = 0
elapsed time = 10
elapsed time = 4
elapsed time = 4
elapsed time = 62
elapsed time = 4
elapsed time = 4
elapsed time = 4

You can also test MimeTypeUtils.parseMimeType

public class ServerApplication {

    public static void main(String[] args) {
        String last = "";
        for (int i = 0; i < 64; i++) {
            last = "video/h" + i;
            MimeTypeUtils.parseMimeType(last);
        }

        long start = System.currentTimeMillis();
        long end;
        for (int i = 0; i < Integer.MAX_VALUE; i++) {
            MimeTypeUtils.parseMimeType(last);
            if (i % 10000 == 0) {
                end = System.currentTimeMillis();
                System.out.println("elapsed time = " + (end - start));
                start = end;
            }
        }
    }
}