I discovered this in a long yak-shaving session migrating an app to Boot 2.4.1 and Hazelcast 4.0.3.
This fails when you run it from the jar:
@SpringBootApplication
public class IssueApplication {
public static void main(String[] args) {
SpringApplication.run(IssueApplication.class, args);
}
@Bean
CommandLineRunner runner(HazelcastInstance hazelcast) {
IMap<String, Foo> sessions = hazelcast.getMap("foos");
return args -> {
sessions.set("foo", new Foo("foo"));
System.err.println(sessions.get("foo"));
sessions.getAsync("foo").whenComplete((u, t) -> System.err.println(u)).toCompletableFuture().get();
};
}
}
class Foo implements Serializable {
private String value;
public Foo() {
}
public Foo(String value) {
this.value = value;
}
public String getValue() {
return this.value;
}
public void setValue(String value) {
this.value = value;
}
@Override
public String toString() {
return "Foo [value=" + this.value + "]";
}
}
It's fine when you run in an IDE, or with java -classpath .... The problem is that Hazelcast uses the Thread.crrentThread().getContextClassLoader() by default, and in the async background thread this is the JarLauncher not the AppClassLoader.
I worked around it with this
@Bean
Config hazelcastConfig() {
Config config = new Config();
config.setClassLoader(SpringApplication.class.getClassLoader());
...
return config;
}
but the issue is more generic really - probably any library that uses JDK fork-join utilities will end up with the same class loader.
Comment From: wilkinsona
The problem is that Hazelcast uses the Thread.crrentThread().getContextClassLoader() by default, and in the async background thread this is the JarLauncher not the AppClassLoader.
I assume that JarLauncher here should be LaunchedURLClassLoader. I think this may also be the wrong way round. In a Boot app you want the TCCL to be the LaunchedURLClassLoader as this is the class loader the can load the application's classes and those of its dependencies. The AppClassLoader can only see the launcher and the JDK.
Unfortunately, the JDK's common fork-join pool uses the app class loader as its TCCL. This makes it unsuitable for use in a packaged Boot application and, I suspect, in any environment with a custom class loader such as a Servlet container. We've seen this problem before (https://github.com/spring-projects/spring-boot/issues/15737) and, in general terms, I don't think we can do anything about it.
In this specific case, it looks like the auto-configuration could be improved by setting the class loader automatically (assuming that's possible while still honouring the rest of the user's configuration).
Comment From: dsyer
Hmm. I think maybe there is still something in Hazelcast which we could ask them to fix.
This app works fine (IDE and JAR) when Hazelcast is not on the classpath:
@SpringBootApplication
public class IssueApplication {
public static void main(String[] args) {
SpringApplication.run(IssueApplication.class, args);
}
@Bean
CommandLineRunner runner() {
return args -> {
ForkJoinPool.commonPool().submit(() -> {
System.err.println(ClassUtils.resolveClassName(Foo.class.getName(), null));
}).get();
};
}
}
...
and then it fails from a JAR if you just put Hazelcast on the classpath but never use it:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-01-14 14:46:30.747 ERROR 1000765 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:807) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:788) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1311) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
at com.example.IssueApplication.main(IssueApplication.java:16) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[hazelcast-issue-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006) ~[na:na]
at com.example.IssueApplication.lambda$runner$1(IssueApplication.java:24) ~[classes!/:0.0.1-SNAPSHOT]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) ~[spring-boot-2.4.2-SNAPSHOT.jar!/:2.4.2-SNAPSHOT]
... 13 common frames omitted
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600) ~[na:na]
... 16 common frames omitted
Caused by: java.lang.IllegalArgumentException: Could not find class [com.example.Foo]
at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:334) ~[spring-core-5.3.3.jar!/:5.3.3]
at com.example.IssueApplication.lambda$runner$0(IssueApplication.java:23) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) ~[na:na]
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[na:na]
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[na:na]
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[na:na]
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[na:na]
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[na:na]
Caused by: java.lang.ClassNotFoundException: com.example.Foo
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.3.3.jar!/:5.3.3]
at org.springframework.util.ClassUtils.resolveClassName(ClassUtils.java:324) ~[spring-core-5.3.3.jar!/:5.3.3]
... 7 common frames omitted
Adding Hazelcast to the classpath (and an empty hazelcast.xml) causes our autoconfiguration to be created, which in turn seems to mess with the default ForkJoinPool.
Comment From: wilkinsona
That's odd. The common pool uses DefaultForkJoinWorkerThreadFactory which "creates a new ForkJoinWorkerThread using the system class loader as the thread context class loader". I don't know how Hazelcast could be messing with that. Sounds like some digging is required.
Comment From: dsyer
This is even weirder. If you have a 2 node cluster it doesn't fail from the JAR. It only fails with one node.
Comment From: dsyer
That's rubbish, sorry. Brain fart probably. It always fails from the JAR for the reason you said (the system class loader is the wrong one).
UPDATE: it sometimes fails. Especially with Java 8 (as opposed to 11, where ForkJoinPool always uses the system class loader).
Comment From: dsyer
So maybe we need to set the class loader in Hazelcast Config in our autoconfig?
Comment From: wilkinsona
Yep, I think so. Now we need to figure out how to do that across all the various configuration options that we support.
Comment From: dsyer
I think they all end up in a Config object somehow. There's a utility in HC, or something, for loading the XML. And the Config is mutable, so we can just create it in whatever way we find the user is trying to suggest and set the class loader. Maybe?
Comment From: HJK181
Yep, I think so. Now we need to figure out how to do that across all the various configuration options that we support.
@wilkinsona any chance to fix 475 in a similar way?
Comment From: wilkinsona
I don't know as I'm not familiar with Feign.
Comment From: dsyer
Feign doesn't have a customization API where you can inject a class loader, so a similar fix to this one is not available. @OlgaMaciaszek suggested an eager initialization alternative (and gave an example of another similar issue being fixed) in the referenced issue. Someone just has to implement it.
Comment From: HJK181
Thx @dsyer I already read about eager loading of feign clients, but it does not seem to solve the issue.
A valid workaround/hack that works for me is this suggestion. However, I can't judge this as my knowledge of Springs ClassLoading is way to limited.