SpringBoot Spring Boot v3.4.0 causes our staging & production environment to hang and time out

Our services work on all Spring Boot versions prior to v3.4.0 and have been for years. v3.4.0 works in our dev environment and we are unable to reproduce what is occurring in staging and production.

We have 4 Spring Boot apps running on GCP

Spring Authorization server (does not seem to be affected)
Spring Boot with webflux doesn't seem to be affected
Two Spring Boot Web MVC apps are affected

Here is what happens once the deployment hits staging / production:

No errors in the startup log
Very quickly, if not immediately, after deployment, requested endpoint starts to hang

GCP metrics (response times goes to 300 seconds and times out after release) Screenshot 2024-11-30 at 15 15 56

MongoDB console (Activity decreases after release) screencapture-cloud-mongodb-v2-590aeb37c0c6e35c7ce2d87f-2024-11-30-15_09_07

Cloud Run startup log that shows Spring Boot starting up error free and goes directly to timing out Screenshot 2024-11-30 at 19 47 40

Maven dependency tree: dependencies.txt.zip

The next step for us would be to turn on logging to high to see anything interesting shows up. Just spent a Saturday afternoon rolling back from production and trying to figure out where it was coming from. At this moment, I am completely clueless.

Any help would be appreciated.

We deploy Spring Boot with mvn spring-boot:build-image and the plugin config looks like this:

<plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>process-aot</id>
                        <configuration>
                            <profiles>local,staging,prod</profiles>
                        </configuration>
                    </execution>
                </executions>
                <configuration>
                    <image>
                        <env>
                            <BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
                            <BPE_APPEND_JAVA_TOOL_OPTIONS>--add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -Dspring.profiles.active=${environment} -Djava.security.manager=allow</BPE_APPEND_JAVA_TOOL_OPTIONS>
                            <BP_JVM_TYPE>JRE</BP_JVM_TYPE>
                            <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
                        </env>
                    </image>
                </configuration>
            </plugin>

Development environment: Mac OS Sequoia Atlas CLI MongoDb 8 Java 23

Providers: MongoDb Atlas MongoDb 8 Google Cloud Platform / Cloud run Java 23

Comment From: bclozel

I think the critical information here is the state of the JVM threads. This would let us know what's preventing the app from serving requests.

Can you capture this information and let us know?

Comment From: bjornharvold

Hi @bclozel,

Thank you for responding as quickly as you did.

I created an endpoint that would return a thread dump like this:

public String getThreadDump() {
        ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
        ThreadInfo[] threadInfos = threadMXBean.dumpAllThreads(true, true);

        return Arrays.stream(threadInfos)
                .map(ThreadInfo::toString)
                .collect(Collectors.joining("\n"));
    }

Immediately after the application started on CloudRun, I was able to collect the first thread dump here: thread-dump.txt

I can hit the unauthenticated endpoint "/" an unlimited number of times without any issues.

After I authenticate with Spring Authentication Server 1.4.0 and try to hit a secure endpoint, the server becomes completely unresponsive and I am no longer able to catch a thread dump and Cloud Run times out the connection after 300 seconds.

At this point, the unauthenticated endpoint "/" is no longer responsive regardless if I hit it unauthenticated or authenticated.

Looks like this applies to our other Spring Boot instances as well... the moment I am authenticated with SAS and then try to call any endpoint... 💥☠️ Server hangs.

There is nothing in any of the instance logs, including SAS, that show any errors or any looping or anything out of the ordinary.

Here's a screenshot of what all the instance logs look like after : Screenshot 2024-12-01 at 12 31 13

I don't see anything out of the ordinary with SAS v1.4.0 release notes in comparison to the SAS version Spring Boot 3.3.6 depends on: https://github.com/spring-projects/spring-authorization-server/releases/tag/1.4.0

I also don't see anything that should concern us with the latest Spring Data MongoDb release: https://github.com/spring-projects/spring-data-mongodb/releases/tag/4.4.0

The app that is unresponsive, still logs MongoDb pings as you can see on this screenshot [as if everything is honky dory]: Screenshot 2024-12-01 at 12 47 40

Let me know how else I can help here. It's not easy capturing a thread dump with Cloud Run when the app is in this, or any, state.

Comment From: bjornharvold

Just to follow up on memory. The instance that hangs runs on 2gb of memory w. 1 CPU [and has been doing so for years].

Here's cloud run monitoring screenshot: screencapture-console-cloud-google-monitoring-dashboards-integration-cloud-run-cloudrun-monitoring-duration-P1D-2024-12-01-12_53_25

Comment From: bclozel

It looks like something is blocking threads or some memory leak/infinite recursion. I’m not a cloud run expert but is there a way to issue a « kill -3 » signal to the container when it’s behaving badly? This should print a thread stack to the console.

The first thread capture doesn’t show anything related to Spring. Maybe you are using Java agents or instrumentation libraries that are not compatible with the latest Spring version?

Comment From: bjornharvold

There is unfortunately no way to do a kill -3 on a Cloud Run instance.

After your last remark I am leaning towards Sentry being the culprit.

https://github.com/spring-io/start.spring.io/issues/1647 https://github.com/getsentry/sentry-java/issues/3941

Will continue my investigation there. Close issue at will.

Cheers Brian 🍻

Comment From: bclozel

I don't think we can track the source of the problem without a snapshot of the java threads when the app is having issues. This could come from any library on your classpath, anny java agent or incompatibility with a remote resource. I haven't seen anything so far pointing to Spring Boot causing issues; we can reopen this issue if we find new information.

I'm not familiar enough with Google Cloud run but not being able to connect to the JVM in any way is quite limiting. Maybe is there a way to configure the instance to open a port and connect a profiler to the running JVM?

Closing this issue for now.

Comment From: wilkinsona

@bjornharvold rather than using an endpoint to trigger the thread dump, perhaps you could temporarily add another thread to your app that does it periodically?

Comment From: bjornharvold

Hi @wilkinsona

That could work. Will give that a go and report back. What would be the best way to handle the dump once done on the Cloud Run instance in your opinion?

Comment From: wilkinsona

Hopefully it can be output to somewhere from which it can then be copied and pasted for further analysis. If you like some help with that analysis, you could paste it here (if it's not too long) or into a gist or the like. It's very unlikely that it'll contain anything that should be kept confidential, but please do double-check first just in case class or method names reveal more than you'd like.

Comment From: bjornharvold

Hi @wilkinsona

We added a 10 min thread dump scheduler where we right to the GCP Cloud Log. Here's it what it came back with after the the Spring Boot app became unresponsive.

{
  "insertId": "67baccba00085db3f7b820c3",
  "jsonPayload": {
    "@timestamp": "2025-02-23T07:22:34.541835117Z",
    "service.version": "30.14.4-SNAPSHOT",
    "process.thread.name": "pool-3-thread-1",
    "log.level": "WARN",
    "process.pid": 1,
    "log.logger": "travel.wink.inventory.ThreadDumpScheduler",
    "message": "Thread Dump at: 2025-02-23T07:22:34.539263210Z\n\n\"Reference Handler\" daemon prio=10 Id=9 RUNNABLE\n\tat java.base@23.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)\n\tat java.base@23.0.2/java.lang.ref.Reference.processPendingReferences(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)\n\n\n\"Finalizer\" daemon prio=8 Id=10 WAITING on java.lang.ref.NativeReferenceQueue$Lock@1bb42068\n\tat java.base@23.0.2/java.lang.Object.wait0(Native Method)\n\t-  waiting on java.lang.ref.NativeReferenceQueue$Lock@1bb42068\n\tat java.base@23.0.2/java.lang.Object.wait(Unknown Source)\n\tat java.base@23.0.2/java.lang.Object.wait(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.NativeReferenceQueue.await(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.ReferenceQueue.remove0(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.NativeReferenceQueue.remove(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)\n\n\n\"Signal Dispatcher\" daemon prio=9 Id=11 RUNNABLE\n\n\n\"Common-Cleaner\" daemon prio=8 Id=16 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@445257f4\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@445257f4\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.ReferenceQueue.await(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.ReferenceQueue.remove0(Unknown Source)\n\tat java.base@23.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)\n\tat java.base@23.0.2/jdk.internal.ref.CleanerImpl.run(Unknown Source)\n\tat java.base@23.0.2/java.lang.Thread.runWith(Unknown Source)\n\t...\n\n\n\"Notification Thread\" daemon prio=9 Id=17 RUNNABLE\n\n\n\"SentryExecutorServiceThreadFactory-0\" daemon prio=5 Id=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da2209a\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da2209a\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"BufferPoolPruner-1-thread-1\" daemon prio=5 Id=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61303550\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61303550\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"cluster-ClusterId{value='67baca5e41e8df745916f9ff', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=27 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bbdfe7e\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bbdfe7e\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca5e41e8df745916f9ff', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=28 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d06ff0a\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d06ff0a\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca5e41e8df745916f9ff', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=29 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51fcb8c0\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51fcb8c0\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"MaintenanceTimer-4-thread-1\" daemon prio=5 Id=30 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31068d1a\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31068d1a\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-3-thread-1\" daemon prio=5 Id=31 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cdb4a\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cdb4a\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-2-thread-1\" daemon prio=5 Id=32 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f389a40\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f389a40\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa00', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=34 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b361fdf\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b361fdf\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa00', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=35 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@edd2dfe\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@edd2dfe\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa00', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=36 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d1811e4\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d1811e4\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"MaintenanceTimer-7-thread-1\" daemon prio=5 Id=37 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16097c9d\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16097c9d\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-6-thread-1\" daemon prio=5 Id=38 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d7a5164\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d7a5164\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-5-thread-1\" daemon prio=5 Id=39 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4aaf2be8\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4aaf2be8\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa01', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=41 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67159468\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67159468\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa01', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3aa5f183\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3aa5f183\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa01', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@452b66c2\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@452b66c2\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"MaintenanceTimer-9-thread-1\" daemon prio=5 Id=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7668e5e4\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7668e5e4\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-10-thread-1\" daemon prio=5 Id=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d958995\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d958995\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-8-thread-1\" daemon prio=5 Id=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28294e4d\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28294e4d\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa02', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36aa30fc\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36aa30fc\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa02', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dcb408\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dcb408\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6041e8df745916fa02', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=50 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16944c47\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16944c47\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"MaintenanceTimer-11-thread-1\" daemon prio=5 Id=51 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@742d23fe\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@742d23fe\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-13-thread-1\" daemon prio=5 Id=52 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1367d057\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1367d057\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-12-thread-1\" daemon prio=5 Id=53 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71f95ac7\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71f95ac7\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6241e8df745916fa03', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55de70ee\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55de70ee\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6241e8df745916fa03', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c49b323\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c49b323\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"cluster-ClusterId{value='67baca6241e8df745916fa03', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017\" daemon prio=5 Id=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3cce789f\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3cce789f\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)\n\tat com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)\n\tat com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)\n\t...\n\n\n\"MaintenanceTimer-14-thread-1\" daemon prio=5 Id=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@355c9e8e\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@355c9e8e\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-15-thread-1\" daemon prio=5 Id=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1323c1cb\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1323c1cb\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"MaintenanceTimer-16-thread-1\" daemon prio=5 Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7212b827\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7212b827\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\t...\n\n\n\"pool-3-thread-1\" prio=5 Id=62 RUNNABLE\n\tat java.management@23.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)\n\tat java.management@23.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)\n\tat java.management@23.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)\n\tat travel.wink.inventory.ThreadDumpScheduler.lambda$startThreadDumpScheduler$1(ThreadDumpScheduler.java:34)\n\tat travel.wink.inventory.ThreadDumpScheduler$$Lambda/0x00007f1519976b68.run(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.FutureTask.runAndReset(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\t...\n\n\tNumber of locked synchronizers = 1\n\t- java.util.concurrent.ThreadPoolExecutor$Worker@d4df4ce\n\n\n\"RateLimitExecutorDelayThread\" daemon prio=5 Id=65 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2fe4edce\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2fe4edce\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)\n\tat com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)\n\t...\n\n\n\"BatchLogRecordProcessor_WorkerThread-1\" daemon prio=5 Id=72 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2786c52b\n\tat java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)\n\t-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2786c52b\n\tat java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ArrayBlockingQueue.poll(Unknown Source)\n\tat io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor$Worker.run(BatchLogRecordProcessor.java:246)\n\tat java.base@23.0.2/java.lang.Thread.runWith(Unknown Source)\n\tat java.base@23.0.2/java.lang.Thread.run(Unknown Source)\n\n\n\"reactor-http-epoll-1\" daemon prio=5 Id=73 RUNNABLE (in native)\n\tat io.netty.channel.epoll.Native.epollWait(Native Method)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:220)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:213)\n\tat io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t...\n\n\n\"reactor-http-epoll-2\" daemon prio=5 Id=74 RUNNABLE (in native)\n\tat io.netty.channel.epoll.Native.epollWait(Native Method)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:220)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:213)\n\tat io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t...\n\n\n\"reactor-http-epoll-3\" daemon prio=5 Id=75 RUNNABLE (in native)\n\tat io.netty.channel.epoll.Native.epollWait(Native Method)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:220)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:213)\n\tat io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t...\n\n\n\"reactor-http-epoll-4\" daemon prio=5 Id=76 RUNNABLE (in native)\n\tat io.netty.channel.epoll.Native.epollWait(Native Method)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:220)\n\tat io.netty.channel.epoll.Native.epollWait(Native.java:213)\n\tat io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t...\n\n\n\"XNIO-1 I/O-1\" prio=5 Id=78 RUNNABLE (in native)\n\tat java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)\n\tat java.base@23.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)\n\tat java.base@23.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)\n\t-  locked sun.nio.ch.Util$2@5e93d92c\n\t-  locked sun.nio.ch.EPollSelectorImpl@1ea62587\n\tat java.base@23.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)\n\tat org.xnio.nio.WorkerThread.run(WorkerThread.java:544)\n\n\n\"XNIO-1 I/O-2\" prio=5 Id=79 RUNNABLE (in native)\n\tat java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)\n\tat java.base@23.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)\n\tat java.base@23.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)\n\t-  locked sun.nio.ch.Util$2@16243100\n\t-  locked sun.nio.ch.EPollSelectorImpl@71a0172\n\tat java.base@23.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)\n\tat org.xnio.nio.WorkerThread.run(WorkerThread.java:563)\n\n\n\"XNIO-1 Accept\" prio=5 Id=80 RUNNABLE (in native)\n\tat java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)\n\tat java.base@23.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)\n\tat java.base@23.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)\n\t-  locked sun.nio.ch.Util$2@4780f204\n\t-  locked sun.nio.ch.EPollSelectorImpl@68256d6e\n\tat java.base@23.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)\n\tat org.xnio.nio.WorkerThread.run(WorkerThread.java:544)\n\n\n\"DestroyJavaVM\" prio=5 Id=82 RUNNABLE\n\n\n\"ForkJoinPool-1-worker-1\" daemon prio=5 Id=84 WAITING on java.lang.VirtualThread@5fbefd5b owned by \"undertow-4\" Id=89\n\tat java.base@23.0.2/jdk.internal.vm.Continuation.run(Unknown Source)\n\t-  waiting on java.lang.VirtualThread@5fbefd5b\n\tat java.base@23.0.2/java.lang.VirtualThread.runContinuation(Unknown Source)\n\tat java.base@23.0.2/java.lang.VirtualThread$$Lambda/0x00007f1519e7bf80.run(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)\n\t...\n\n\n\"ForkJoinPool-1-worker-2\" daemon prio=5 Id=86 WAITING on java.lang.VirtualThread@8f8764f owned by \"undertow-5\" Id=90\n\tat java.base@23.0.2/jdk.internal.vm.Continuation.run(Unknown Source)\n\t-  waiting on java.lang.VirtualThread@8f8764f\n\tat java.base@23.0.2/java.lang.VirtualThread.runContinuation(Unknown Source)\n\tat java.base@23.0.2/java.lang.VirtualThread$$Lambda/0x00007f1519e7bf80.run(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)\n\tat java.base@23.0.2/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)\n\t...\n\n\n\"MasterPoller\" daemon prio=5 Id=91 RUNNABLE (in native)\n\tat java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)\n\tat java.base@23.0.2/sun.nio.ch.EPollPoller.poll(Unknown Source)\n\tat java.base@23.0.2/sun.nio.ch.Poller.pollerLoop(Unknown Source)\n\tat java.base@23.0.2/sun.nio.ch.Poller$Pollers$$Lambda/0x00007f1519f63ea0.run(Unknown Source)\n\tat java.base@23.0.2/java.lang.Thread.runWith(Unknown Source)\n\tat java.base@23.0.2/java.lang.Thread.run(Unknown Source)\n\tat java.base@23.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)\n\n\n",
    "ecs.version": "8.11"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "service_name": "inventory",
      "location": "us-central1",
      "revision_name": "inventory-01294-zdj",
      "configuration_name": "inventory",
      "project_id": "staging-iko-travel"
    }
  },
  "timestamp": "2025-02-23T07:22:34.548275Z",
  "labels": {
    "instanceId": "00fd7d733704c69872406649fee5b04c2a158aa21d128b56fe1dd07b04122a7714b920b2c3ae8cd4e85dc009925ba8ab5f7579740f22df31cae8a9ccdeb9fc3a776647f88f"
  },
  "logName": "projects/staging-iko-travel/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2025-02-23T07:22:34.554067263Z",
  "errorGroups": [
    {
      "id": "CM7-zdLUtfTL-wE"
    }
  ]
}

Chat-GPT came back with this response:

Your Spring Boot application appears to be unresponsive due to several potential issues based on the thread dump. Here are the key observations:

Potential Causes of Unresponsiveness

Blocked MongoDB Connection Threads

Multiple threads are stuck in TIMED_WAITING in DefaultServerMonitor.waitForSignalOrTimeout(). This suggests that your application is waiting on MongoDB responses that might not be coming.
This could indicate:
Network issues with MongoDB (iko-staging-shard on [gcp.mongodb.net](http://gcp.mongodb.net/)).
A slow or overloaded MongoDB server.
Connection pooling exhaustion.
Thread Pool Starvation

Many threads are in TIMED_WAITING on ScheduledThreadPoolExecutor$DelayedWorkQueue.take(). This means tasks are waiting for execution but not being picked up.
It suggests:
A potential deadlock in your application logic.
Overloaded thread pools, possibly due to too many blocking calls.
Reactor Netty (Web Server) is Active

reactor-http-epoll-* threads are RUNNABLE, meaning the Netty server is running.
However, if requests are stuck waiting for MongoDB responses, your app may not be able to process new requests effectively.
Rate-Limiting Executor is Waiting

RateLimitExecutorDelayThread is in WAITING, indicating a rate limiter is actively delaying tasks.
If this is used to manage API requests or database access, it might be configured too strictly.
ForkJoinPool Virtual Threads Are Blocked

The ForkJoinPool workers are WAITING on VirtualThread.runContinuation(), meaning some parallel tasks are waiting on locks or external I/O.
This could cause slow processing of concurrent tasks.
Recommended Actions

MongoDB Connection Issues

Check MongoDB availability: ping [iko-staging-shard-00-01.xyhwi.gcp.mongodb.net](http://iko-staging-shard-00-01.xyhwi.gcp.mongodb.net/)
Ensure MongoDB is reachable from your service.
Review connection pool settings: Increase maxPoolSize in MongoDB client settings.
Enable logging for slow queries in MongoDB.
Thread Pool Adjustments

Increase core pool size for ScheduledThreadPoolExecutor.
Reduce blocking operations inside scheduled tasks.
Rate-Limiting Fix

If you're using a rate limiter, check if limits are set too low.
General Debugging

Use JVM profiling tools (jstack, VisualVM, Flight Recorder) to analyze active threads.
Add timeouts on MongoDB queries (find().maxTime(10, TimeUnit.SECONDS)).
Would you like help with specific configuration changes?

Comment From: bjornharvold

Why would Spring Data MongoDb have issues with Spring Boot 3.4.x?
I see there is also a TIMED_WAITING on SentryExecutorServiceThreadFactory

Any thoughts on this would be appreciated. Our runs great on 3.3.x. and has been for the entirety of 3.3.x and 3.2.x.

🍻

Comment From: wilkinsona

Reformatted thread dump:


"Reference Handler" daemon prio=10 Id=9 RUNNABLE
    at java.base@23.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
    at java.base@23.0.2/java.lang.ref.Reference.processPendingReferences(Unknown Source)
    at java.base@23.0.2/java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

"Finalizer" daemon prio=8 Id=10 WAITING on java.lang.ref.NativeReferenceQueue$Lock@1bb42068
    at java.base@23.0.2/java.lang.Object.wait0(Native Method)
    -  waiting on java.lang.ref.NativeReferenceQueue$Lock@1bb42068
    at java.base@23.0.2/java.lang.Object.wait(Unknown Source)
    at java.base@23.0.2/java.lang.Object.wait(Unknown Source)
    at java.base@23.0.2/java.lang.ref.NativeReferenceQueue.await(Unknown Source)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.remove0(Unknown Source)
    at java.base@23.0.2/java.lang.ref.NativeReferenceQueue.remove(Unknown Source)
    at java.base@23.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Signal Dispatcher" daemon prio=9 Id=11 RUNNABLE

"Common-Cleaner" daemon prio=8 Id=16 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@445257f4
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@445257f4
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.await(Unknown Source)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.remove0(Unknown Source)
    at java.base@23.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)
    at java.base@23.0.2/jdk.internal.ref.CleanerImpl.run(Unknown Source)
    at java.base@23.0.2/java.lang.Thread.runWith(Unknown Source)
    ...

"Notification Thread" daemon prio=9 Id=17 RUNNABLE

"SentryExecutorServiceThreadFactory-0" daemon prio=5 Id=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da2209a
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4da2209a
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"BufferPoolPruner-1-thread-1" daemon prio=5 Id=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61303550
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61303550
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"cluster-ClusterId{value='67baca5e41e8df745916f9ff', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=27 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bbdfe7e
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bbdfe7e
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca5e41e8df745916f9ff', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=28 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d06ff0a
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d06ff0a
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca5e41e8df745916f9ff', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=29 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51fcb8c0
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51fcb8c0
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"MaintenanceTimer-4-thread-1" daemon prio=5 Id=30 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31068d1a
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31068d1a
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-3-thread-1" daemon prio=5 Id=31 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cdb4a
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cdb4a
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-2-thread-1" daemon prio=5 Id=32 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f389a40
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f389a40
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa00', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=34 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b361fdf
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b361fdf
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa00', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=35 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@edd2dfe
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@edd2dfe
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa00', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=36 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d1811e4
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d1811e4
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"MaintenanceTimer-7-thread-1" daemon prio=5 Id=37 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16097c9d
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16097c9d
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-6-thread-1" daemon prio=5 Id=38 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d7a5164
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d7a5164
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-5-thread-1" daemon prio=5 Id=39 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4aaf2be8
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4aaf2be8
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa01', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=41 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67159468
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67159468
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa01', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3aa5f183
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3aa5f183
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa01', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@452b66c2
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@452b66c2
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...


"MaintenanceTimer-9-thread-1" daemon prio=5 Id=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7668e5e4
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7668e5e4
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-10-thread-1" daemon prio=5 Id=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d958995
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d958995
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-8-thread-1" daemon prio=5 Id=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28294e4d
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28294e4d
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa02', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36aa30fc
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36aa30fc
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa02', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dcb408
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dcb408
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6041e8df745916fa02', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=50 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16944c47
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16944c47
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"MaintenanceTimer-11-thread-1" daemon prio=5 Id=51 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@742d23fe
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@742d23fe
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-13-thread-1" daemon prio=5 Id=52 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1367d057
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1367d057
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-12-thread-1" daemon prio=5 Id=53 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71f95ac7
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71f95ac7
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"cluster-ClusterId{value='67baca6241e8df745916fa03', description='null'}-iko-staging-shard-00-01.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55de70ee
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55de70ee
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6241e8df745916fa03', description='null'}-iko-staging-shard-00-02.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c49b323
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c49b323
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"cluster-ClusterId{value='67baca6241e8df745916fa03', description='null'}-iko-staging-shard-00-00.xyhwi.gcp.mongodb.net:27017" daemon prio=5 Id=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3cce789f
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3cce789f
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lambda$waitForSignalOrTimeout$1(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor$$Lambda/0x00007f1519852998.get(Unknown Source)
    at com.mongodb.internal.Locks.checkedWithLock(Locks.java:62)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForSignalOrTimeout(DefaultServerMonitor.java:345)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.waitForNext(DefaultServerMonitor.java:331)
    ...

"MaintenanceTimer-14-thread-1" daemon prio=5 Id=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@355c9e8e
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@355c9e8e
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-15-thread-1" daemon prio=5 Id=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1323c1cb
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1323c1cb
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"MaintenanceTimer-16-thread-1" daemon prio=5 Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7212b827
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7212b827
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ...

"pool-3-thread-1" prio=5 Id=62 RUNNABLE
    at java.management@23.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
    at java.management@23.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
    at java.management@23.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
    at travel.wink.inventory.ThreadDumpScheduler.lambda$startThreadDumpScheduler$1(ThreadDumpScheduler.java:34)
    at travel.wink.inventory.ThreadDumpScheduler$$Lambda/0x00007f1519976b68.run(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    ...

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@d4df4ce

"RateLimitExecutorDelayThread" daemon prio=5 Id=65 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2fe4edce
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2fe4edce
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinPool.managedBlock(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
    at com.google.maps.internal.RateLimitExecutorService.run(RateLimitExecutorService.java:78)
    ...

"BatchLogRecordProcessor_WorkerThread-1" daemon prio=5 Id=72 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2786c52b
    at java.base@23.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2786c52b
    at java.base@23.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ArrayBlockingQueue.poll(Unknown Source)
    at io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor$Worker.run(BatchLogRecordProcessor.java:246)
    at java.base@23.0.2/java.lang.Thread.runWith(Unknown Source)
    at java.base@23.0.2/java.lang.Thread.run(Unknown Source)

"reactor-http-epoll-1" daemon prio=5 Id=73 RUNNABLE (in native)
    at io.netty.channel.epoll.Native.epollWait(Native Method)
    at io.netty.channel.epoll.Native.epollWait(Native.java:220)
    at io.netty.channel.epoll.Native.epollWait(Native.java:213)
    at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ...

"reactor-http-epoll-2" daemon prio=5 Id=74 RUNNABLE (in native)
    at io.netty.channel.epoll.Native.epollWait(Native Method)
    at io.netty.channel.epoll.Native.epollWait(Native.java:220)
    at io.netty.channel.epoll.Native.epollWait(Native.java:213)
    at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ...

"reactor-http-epoll-3" daemon prio=5 Id=75 RUNNABLE (in native)
    at io.netty.channel.epoll.Native.epollWait(Native Method)
    at io.netty.channel.epoll.Native.epollWait(Native.java:220)
    at io.netty.channel.epoll.Native.epollWait(Native.java:213)
    at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ...

"reactor-http-epoll-4" daemon prio=5 Id=76 RUNNABLE (in native)
    at io.netty.channel.epoll.Native.epollWait(Native Method)
    at io.netty.channel.epoll.Native.epollWait(Native.java:220)
    at io.netty.channel.epoll.Native.epollWait(Native.java:213)
    at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:308)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:365)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ...

"XNIO-1 I/O-1" prio=5 Id=78 RUNNABLE (in native)
    at java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)
    at java.base@23.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
    at java.base@23.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
    -  locked sun.nio.ch.Util$2@5e93d92c
    -  locked sun.nio.ch.EPollSelectorImpl@1ea62587
    at java.base@23.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:544)

"XNIO-1 I/O-2" prio=5 Id=79 RUNNABLE (in native)
    at java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)
    at java.base@23.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
    at java.base@23.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
    -  locked sun.nio.ch.Util$2@16243100
    -  locked sun.nio.ch.EPollSelectorImpl@71a0172
    at java.base@23.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:563)

"XNIO-1 Accept" prio=5 Id=80 RUNNABLE (in native)
    at java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)
    at java.base@23.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
    at java.base@23.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
    -  locked sun.nio.ch.Util$2@4780f204
    -  locked sun.nio.ch.EPollSelectorImpl@68256d6e
    at java.base@23.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:544)

"DestroyJavaVM" prio=5 Id=82 RUNNABLE

"ForkJoinPool-1-worker-1" daemon prio=5 Id=84 WAITING on java.lang.VirtualThread@5fbefd5b owned by "undertow-4" Id=89
    at java.base@23.0.2/jdk.internal.vm.Continuation.run(Unknown Source)
    -  waiting on java.lang.VirtualThread@5fbefd5b
    at java.base@23.0.2/java.lang.VirtualThread.runContinuation(Unknown Source)
    at java.base@23.0.2/java.lang.VirtualThread$$Lambda/0x00007f1519e7bf80.run(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    ...

"ForkJoinPool-1-worker-2" daemon prio=5 Id=86 WAITING on java.lang.VirtualThread@8f8764f owned by "undertow-5" Id=90
    at java.base@23.0.2/jdk.internal.vm.Continuation.run(Unknown Source)
    -  waiting on java.lang.VirtualThread@8f8764f
    at java.base@23.0.2/java.lang.VirtualThread.runContinuation(Unknown Source)
    at java.base@23.0.2/java.lang.VirtualThread$$Lambda/0x00007f1519e7bf80.run(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base@23.0.2/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    ...

"MasterPoller" daemon prio=5 Id=91 RUNNABLE (in native)
    at java.base@23.0.2/sun.nio.ch.EPoll.wait(Native Method)
    at java.base@23.0.2/sun.nio.ch.EPollPoller.poll(Unknown Source)
    at java.base@23.0.2/sun.nio.ch.Poller.pollerLoop(Unknown Source)
    at java.base@23.0.2/sun.nio.ch.Poller$Pollers$$Lambda/0x00007f1519f63ea0.run(Unknown Source)
    at java.base@23.0.2/java.lang.Thread.runWith(Unknown Source)
    at java.base@23.0.2/java.lang.Thread.run(Unknown Source)
    at java.base@23.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)

Comment From: wilkinsona

Spring Boot 3.3.x uses 5.0.x of the MongoDB Java driver and Spring Boot 3.4.x uses 5.2.x. Given that upgrade and the large number of Mongo-related threads in the dump that are waiting for something, it does indeed look like a Mongo-related problem. Neither Spring Boot nor Spring Data MongoDB are involved with Mongo at this low level and, in your position, I'd continue the investigation working on the assumption that it's a Mongo Java Driver problem. It appears they've had a similar problem in the past.

One thing that's not clear from the dump is why the server's become unresponsive. I can't see any web request threads stuck waiting for something. Perhaps there wasn't a stuck request at the time of the dump? Regardless, I think addressing the apparent problem with Mongo should be the next step – it'll either solve the problem or eliminate a possible cause.

Comment From: bjornharvold

Thank you @wilkinsona 🦸‍♂️

To confirm, our server had become unresponsive. It would time out on all our endpoints; even simple /ping ones.
The upgrade to 3.4.x works locally and always has.
We use MongoDb Atlas CLI to stand up a local MongoDb instance so the differences should be negligible.
The 6 MongoDb threads are connections to 6 different MongoDb databases
The timeout only occurs on Web MVC and not WebFlux
Sometimes we cannot replicate the timeout immediately. We just stood up a new instance in our staging environment with Spring Boot 3.4.3 and what timed out yesterday is now working

We will need to keep on investigating. 😑

Comment From: wilkinsona

If it's not Mongo that's the problem, then I would turn attention to the web server. Looking at the thread dump, you appear to be using Undertow (the XNIO threads). Have you tried with Tomcat or Jetty? If the problem doesn't occur when using WebFlux (and assuming that you're using the default of Netty there), that may be another pointer towards an Undertow problem.

Comment From: bjornharvold

This seems to have resolved itself with Spring Boot v3.4.3 🤔.

Nothing in the release notes that really stands out. At least no Undertow upgrade.

Staging has not become unresponsive in the last 24 hours.

We will keep on testing and report back.

Comment From: bjornharvold

I can confirm that it has not been resolved yet. Just deployed to production w v3.4.3. Spring Authorization Server instance is running fine. Our webflux instance runs fine. Our 2 Spring Web MVC instances that see the most traffic both started to hang out some point after release. One was complaining about OutOfMemory so we upgraded from 2Gb to 8Gb. Instance starts up and becomes unresponsive immediately so not a memory issue. We also replicated locally with the same amount of available server memory to rule that out a few weeks ago.

Where we are at: - We have no issues running the Spring Boot apps locally - Spring WebFlux is fine - It's not a memory issue - We cannot collect a valuable thread dump as the server says it hasn't hung => so waiting on something - It still fails randomly which indicates that at some point it tries to execute some code that then makes the whole server become unresponsive.

This all equates [approx] to 🫤

Comment From: wilkinsona

We cannot collect a valuable thread dump as the server says it hasn't hung => so waiting on something

I am intrigued by this. How has the server identified that it hasn't hung and how are you being made aware of that?

Comment From: bjornharvold

Hi @wilkinsona

The last thread dump we shared was taken after the system had become unresponsive and there is nothing in there that implies it's hung.

Also, the fact that we were able to take a thread dump from a scheduled task (not endpoint) shows that the instance is not completely unresponsive.

That's how we came to this conclusion.

Will now try to use Tomcat as you suggested earlier and report back.

Comment From: wilkinsona

Thanks, I'm with you now. Given that the server's threads show no signs of being unable to handle an HTTP request, I also wonder if there's anything in front of the server that could be causing the problem. Perhaps it's only present in your staging and production environments which might explain why the dev environment is unaffected. It would hopefully be informative to know exactly where the requests are getting stuck. It doesn't look like it's in the server so, presumably, it's somewhere between the client and the server. Perhaps Google support can offer some guidance on diagnosing such a problem.

Comment From: bjornharvold

...but why would Spring Boot v3.4.x be responsible for anything Google or Cloudflare does? V3.3.x working like a champ.

Comment From: wilkinsona

Something may have changed how the connections are managed. For example, perhaps connections are being kept alive now when previously they were not.

Comment From: bjornharvold

We released a Tomcat version to staging environment. Experiencing the same unresponsiveness.

Spring Web MVC App 1 became unresponsive immediately on start:

Spring Web MVC App 2 still running. If we released this to production, it would go unresponsive shortly there after. It seems there is "something" that occurs after startup (either immediately or later) that causes this to happen.