SpringBoot ClassNotFoundException is thrown when loading protocol resolvers from ForkJoinPool task

Hi, we have upgraded Spring Boot app version from 3.3.3 to 3.3.4 and now the SSL bundles cannot be loaded. See the full stack trace of the exception below.

I've found there have been some changes related to SSL bundles in #42119. I can see the key stores are being lazily loaded but not sure how (and if) it can change the class loader that doesn't see Base64ProtocolResolver.

We have integration @SpringBootTest tests that use the SSL bundles and they are successfully passing. So most probably the app must be compiled and packaged as we see those exceptions only when we run the app (built as a JAR) in Docker container. But I could still be missing something.

When I downgrade to 3.3.3, it works OK as expected.

I don't have a sample application that reproduces the issue yet.

The app runs on reactive stack.

java.lang.IllegalStateException: Unable to create key store: Could not load store from 'classpath:{CERT}.p12'
    at org.springframework.boot.ssl.jks.JksSslStoreBundle.createKeyStore(JksSslStoreBundle.java:96)
    at org.springframework.boot.ssl.jks.JksSslStoreBundle.lambda$new$0(JksSslStoreBundle.java:59)
    at org.springframework.util.function.SingletonSupplier.get(SingletonSupplier.java:106)
    at org.springframework.boot.ssl.jks.JksSslStoreBundle.getKeyStore(JksSslStoreBundle.java:65)
    at org.springframework.boot.ssl.DefaultSslManagerBundle.getKeyManagerFactory(DefaultSslManagerBundle.java:45)
    at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory$SslConfigurer.customizeSsl(ReactorClientHttpConnectorFactory.java:91)
    at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83)
    at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60)
    at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88)
    at reactor.netty.http.client.HttpClient.secure(HttpClient.java:1430)
    at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory$SslConfigurer.configure(ReactorClientHttpConnectorFactory.java:84)
    at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorNettyHttpClientMapper.lambda$of$1(ReactorNettyHttpClientMapper.java:65)
    at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
    at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
    at org.springframework.http.client.reactive.ReactorClientHttpConnector.createHttpClient(ReactorClientHttpConnector.java:125)
    at org.springframework.http.client.reactive.ReactorClientHttpConnector.connect(ReactorClientHttpConnector.java:142)
    at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.exchange(ExchangeFunctions.java:102)
    at org.springframework.web.reactive.function.client.DefaultWebClient$ObservationFilterFunction.filter(DefaultWebClient.java:737)
    at org.springframework.web.reactive.function.client.ExchangeFilterFunction.lambda$apply$2(ExchangeFilterFunction.java:73)
    at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultRequestBodyUriSpec.lambda$exchange$11(DefaultWebClient.java:457)
    at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:47)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:210)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
    at reactor.core.publisher.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
    at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
    at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.request(FluxMapFuseable.java:360)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
    at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.request(FluxOnErrorReturn.java:153)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:968)
    at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onSubscribe(FluxOnErrorReturn.java:95)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onSubscribe(FluxMapFuseable.java:265)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
    at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
    at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)
    at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430)
    at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
    at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
    at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
    at reactor.core.publisher.FluxConcatMap$WeakScalarSubscription.request(FluxConcatMap.java:480)
    at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
    at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:202)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
    at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
    at reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    at oracle.r2dbc.impl.OracleReactiveJdbcAdapter$1.onNext(OracleReactiveJdbcAdapter.java:783)
    at oracle.r2dbc.impl.AsyncLock$UsingConnectionSubscriber.onNext(AsyncLock.java:481)
    at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
    at org.reactivestreams.FlowAdapters$FlowToReactiveSubscriber.onNext(FlowAdapters.java:211)
    at oracle.jdbc.driver.PhasedPublisher$PhasedSubscription.lambda$emitNextItem$0(PhasedPublisher.java:403)
    at oracle.jdbc.driver.PhasedPublisher.handleOnNext(PhasedPublisher.java:267)
    at oracle.jdbc.driver.PhasedPublisher$PhasedSubscription.lambda$emitNextItem$1(PhasedPublisher.java:401)
    at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Could not load store from 'classpath:{CERT}.p12'
    at org.springframework.boot.ssl.jks.JksSslStoreBundle.loadKeyStore(JksSslStoreBundle.java:125)
    at org.springframework.boot.ssl.jks.JksSslStoreBundle.createKeyStore(JksSslStoreBundle.java:91)
    ... 88 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unable to instantiate factory class [org.springframework.boot.io.Base64ProtocolResolver] for factory type [org.springframework.core.io.ProtocolResolver]
    at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$throwing$0(SpringFactoriesLoader.java:647)
    at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$handleMessage$3(SpringFactoriesLoader.java:671)
    at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:231)
    at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:206)
    at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:142)
    at org.springframework.boot.io.ApplicationResourceLoader.<init>(ApplicationResourceLoader.java:53)
    at org.springframework.boot.io.ApplicationResourceLoader.<init>(ApplicationResourceLoader.java:41)
    at org.springframework.boot.ssl.jks.JksSslStoreBundle.loadKeyStore(JksSslStoreBundle.java:119)
    ... 89 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.io.Base64ProtocolResolver
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
    at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Unknown Source)
    at java.base/java.lang.Class.forName(Unknown Source)
    at org.springframework.util.ClassUtils.forName(ClassUtils.java:304)
    at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:224)
    ... 94 common frames omitted

Comment From: wilkinsona

Thanks for the report. The ApplicationResourceLoader is being created in such a way that the thread context class loader will be used to load its protocol resolvers. It looks like this is ClassLoaders$AppClassLoader which, if you're running your application with java -jar or java -cp … org.springframework.boot.loader.launch.JarLauncher won't have org.springframework.boot.io.Base64ProtocolResolver on its classpath. It sounds like you're using java -jar in your Docker container. Can you confirm that's the case?

In the absence of a sample that reproduces the problem, you could test the above theory by patching line 119 of JksSslStoreBundle:

Resource resource = new ApplicationResourceLoader(getClass().getClassLoader()).getResource(location);

Comment From: raestio

Thank you for the hint. Our Dockerfile looked like this:

docs Spring Boot 3.2.0 - Dockerfiles

...
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.launch.JarLauncher"]

So java org.springframework.boot.loader.launch.JarLauncher.

But I have updated the Dockerfile based on the latest docs Spring Boot 3.3.4 - Dockerfiles / Source and the issue is gone, works OK. Thank you for the help!

Now I'm thinking, I can see the docs for the tools mode were updated in version 3.3.0 - #40094 - which I missed. I believe this feature https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#cds-support was the related one. Does it mean the changes made for the CDS feature caused some incompatibility with the launching the extracted JAR in an older way before 3.3.0 as I had it before? Because now I'm still running the extracted JAR but with java -jar application.jar (as stated in the latest docs) and it works. I'm just trying to piece together the points from your comment and find out why it works. I guess a different class loader is used.

Comment From: wilkinsona

I guess a different class loader is used.

Yes, that's right. Since 3.3, the extracted jar uses a Class-Path manifest header to reference all of the application's dependencies so everything's loaded by the app class loader. As you suspected, this was primarily for CDS where taking Spring Boot's custom class loader out of the picture significantly improves the hit rate against the CDS archive and increases the startup time benefits.

Comment From: wilkinsona

There are several places in 3.3.x that are susceptible to this problem:

https://github.com/spring-projects/spring-boot/blob/da1edd3833e8d99b2f85855f09864ee6d9880fa4/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/convert/StringToFileConverter.java#L37

https://github.com/spring-projects/spring-boot/blob/da1edd3833e8d99b2f85855f09864ee6d9880fa4/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/ssl/jks/JksSslStoreBundle.java#L119

https://github.com/spring-projects/spring-boot/blob/da1edd3833e8d99b2f85855f09864ee6d9880fa4/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/ssl/pem/PemContent.java#L122

https://github.com/spring-projects/spring-boot/blob/da1edd3833e8d99b2f85855f09864ee6d9880fa4/spring-boot-project/spring-boot-autoconfigure/src/main/java/org/springframework/boot/autoconfigure/ssl/BundleContentProperty.java#L73

In addition, on main, there is also the following:

https://github.com/spring-projects/spring-boot/blob/f908fcd1f3d962d8d8fb930b622d61d009e31fda/spring-boot-project/spring-boot-autoconfigure/src/main/java/org/springframework/boot/autoconfigure/couchbase/CouchbaseAutoConfiguration.java#L120

I think we should probably change each of them to use getClass().getClassLoader() as the resource loader's class loader but I'd like to review them with the rest of the team in case I'm overlooking something.

Comment From: philwebb

Looking into this and I wonder if we should have a ApplicationResourceLoader constructor that allows us to specify a different classloader for the SpringFactoriesLoader? I think that might help bring StringToFileConverter at least closer to the older code.

For this failure specifically, I feel like we somehow need to set the classloader that gets used on the actual SslBundle.

Comment From: philwebb

https://github.com/philwebb/gh-42468 reproduces the issue (in a slightly different form)

Comment From: philwebb

I've created #42838 to consider the broader classloader issue so we can target a focused fix for this specific problem in 3.3.x.

Comment From: philwebb

Reopening because I missed the BundleContentProperty class