To build an executable war(with jsp) OCI image, when using tomcat-embed-jasper dependency, the spring-boot-maven-plugin build-image is OK, but can't run successfully.
I dig a little deeper, during the build-image phase, spring-boot-maven-plugin using buidpacks add a soft link of spring-cloud-bindings-1.7.1.jar to /WEB-INF/lib folder, this jar cause tomcat-embed-jasper's StandardJarScanner throw a NullPointerException.
I found a solution then,
Add below in application.yml to skip the scan.
server:
tomcat:
additional-tld-skip-patterns:
- spring-cloud-bindings*.jar
The example code is in the repo zhanming/demo
If comment this configuration out, the issue will recur.
Is this soft link of spring-cloud-bindings-1.7.1.jar necessary?
Comment From: wilkinsona
Thanks for the report. Spring Cloud Bindings is added to the image by the Spring Boot Buildpack and is out of Spring Boot's control. There is an environment variable, BPL_SPRING_CLOUD_BINDINGS_ENABLED, that can be used to control whether or not the bindings are available but it don't believe it has an effect on whether they're contributed to the image. If you'd like such an option to be added, please raise an issue here.
Interestingly, I can't reproduce the problem in an app with an explicit dependency on spring-cloud-bindings. Tomcat is able to scan the jar when launched in an IDE, using java -jar app.war, and unpacked using java org.springframework.boot.loader.WarLauncher. We'll need to investigate further to see what's causing it to fail in an image.
Comment From: zhanming
Thanks for the quick feedback.
I think spring-cloud-bindings is good. The issue is in spring-boot-maven-plugin's build-image phase, as you said Spring Boot Buildpack add a soft link of spring-cloud-bindings to /WEB-INF/lib
Follow the Whats New in Spring Boot 2.3
BTW: Thanks to Phil Webb, I'm hoping What's New in Spring Boot 2.5
You can use dive to check the image.
First, build the image.
mvn clean spring-boot:build-image
Install dive on macOS
$ brew install dive
Then dive it.
dive demo:0.0.1-SNAPSHOT
In /workspace/WEB-INF/lib folder, you can see spring-cloud-bindings is a soft link in there.
I will raise an issue to Spring Boot Buildpack, thanks.
Comment From: wilkinsona
Thanks. It's the fact that it's a symlink that is key. I can reproduce the problem locally with an unpacked war and spring-cloud-bindings symlinked into WEB-INF/lib. The same problem also occurs with standalone Tomcat:
wget https://apache.mirrors.nublue.co.uk/tomcat/tomcat-9/v9.0.48/bin/apache-tomcat-9.0.48.tar.gz
tar -xzf apache-tomcat-9.0.48.tar.gz
mkdir -p apache-tomcat-9.0.48/webapps/symlink-problem/WEB-INF/lib
wget https://repo.spring.io/artifactory/libs-release-local/org/springframework/cloud/spring-cloud-bindings/1.7.1/spring-cloud-bindings-1.7.1.jar
ln -s spring-cloud-bindings-1.7.1.jar apache-tomcat-9.0.48/webapps/symlink-problem/WEB-INF/lib
apache-tomcat-9.0.48/bin/catalina.sh run
24-Jun-2021 10:26:55.777 SEVERE [main] org.apache.catalina.startup.HostConfig.deployDirectory Error deploying web application directory [/Users/awilkinson/dev/temp/tomcat-symlink-problem/apache-tomcat-9.0.48/webapps/symlink-problem]
java.lang.IllegalStateException: Error starting child
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:731)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1185)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1933)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1095)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1618)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/symlink-problem]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728)
... 37 more
Caused by: java.lang.NullPointerException
at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:382)
at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:195)
at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:2136)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1289)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:986)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:303)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5135)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 38 more
@markt-asf This looks like a Tomcat bug to me. If you agree, would you like us to open a Tomcat issue?
Comment From: markt-asf
Generally, symlinks won't work unless allowLinking is set to true on Context.getResources(). This looks like a symptom of using a symlink when that isn't set but it also looks like Tomcat could handle this better. To be consistent with how Tomcat handles this elsewhere, I'd expect Tomcat to behave as if the symlinked JAR wasn't present. Please do open a Tomcat issue for this.
Comment From: wilkinsona
Thanks, Mark. I've opened https://bz.apache.org/bugzilla/show_bug.cgi?id=65397.