SpringBoot Exploded archives launched using the launcher have an incorrectly ordered classpath that also contains non-existent jar files

I have a boot application which is executing flyway migrations on app start. The migration itself runs succesfully. But when create a docker image with mvn spring-boot:build-image and run the image its show many warnings in the log:

Skipping unloadable jar file: file:/workspace/XXXjar

Im also try to create a custom image following the instruction here https://spring.io/blog/2020/08/14/creating-efficient-docker-images-with-spring-boot-2-3. I have placed the app code in the folder /opt/app inside of the container. Here im also getting this warnings: Skipping unloadable jar file: file:/opt/app/XXX.jar (/opt/app/XXX.jar) Here i found out that the scanner try to upen the jar in the wrong folder. Its not placed in /opt/app its inside of /opt/app/BOOT-INF/lib/XXXjar.

With fat jar images its working without warnings.

The migration script are placed in the deafult folder classpath:db/migrations

Spring Boot Version: 2.4.0

Comment From: wilkinsona

Thanks for the report. I've reproduced the problem.

Comment From: wilkinsona

The underlying problem exists in Boot 2.3.x. The warning messages only appear with Boot 2.4 due to some changes in Flyway 7. With Spring Boot 2.3 and 2.4, the LaunchedURLClassLoader ends up with two entries for each jar file:

file:/workspace/BOOT-INF/classes/
file:/workspace/BOOT-INF/lib/HikariCP-3.4.5.jar
file:/workspace/BOOT-INF/lib/flyway-core-6.4.4.jar
file:/workspace/BOOT-INF/lib/h2-1.4.200.jar
file:/workspace/BOOT-INF/lib/jackson-annotations-2.11.3.jar
file:/workspace/BOOT-INF/lib/jackson-core-2.11.3.jar
file:/workspace/BOOT-INF/lib/jackson-databind-2.11.3.jar
file:/workspace/BOOT-INF/lib/jackson-datatype-jdk8-2.11.3.jar
file:/workspace/BOOT-INF/lib/jackson-datatype-jsr310-2.11.3.jar
file:/workspace/BOOT-INF/lib/jackson-module-parameter-names-2.11.3.jar
file:/workspace/BOOT-INF/lib/jakarta.annotation-api-1.3.5.jar
file:/workspace/BOOT-INF/lib/jakarta.el-3.0.3.jar
file:/workspace/BOOT-INF/lib/jul-to-slf4j-1.7.30.jar
file:/workspace/BOOT-INF/lib/log4j-api-2.13.3.jar
file:/workspace/BOOT-INF/lib/log4j-to-slf4j-2.13.3.jar
file:/workspace/BOOT-INF/lib/logback-classic-1.2.3.jar
file:/workspace/BOOT-INF/lib/logback-core-1.2.3.jar
file:/workspace/BOOT-INF/lib/slf4j-api-1.7.30.jar
file:/workspace/BOOT-INF/lib/snakeyaml-1.26.jar
file:/workspace/BOOT-INF/lib/spring-aop-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-beans-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-autoconfigure-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-starter-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-starter-jdbc-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-starter-json-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-starter-logging-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-starter-tomcat-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-boot-starter-web-2.3.6.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-cloud-bindings-1.6.0.jar
file:/workspace/BOOT-INF/lib/spring-context-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-core-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-expression-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-jcl-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-jdbc-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-tx-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-web-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/spring-webmvc-5.2.11.RELEASE.jar
file:/workspace/BOOT-INF/lib/tomcat-embed-core-9.0.39.jar
file:/workspace/BOOT-INF/lib/tomcat-embed-websocket-9.0.39.jar
file:/workspace/spring-boot-starter-jdbc-2.3.6.RELEASE.jar
file:/workspace/spring-boot-starter-2.3.6.RELEASE.jar
file:/workspace/spring-boot-2.3.6.RELEASE.jar
file:/workspace/spring-boot-autoconfigure-2.3.6.RELEASE.jar
file:/workspace/spring-boot-starter-logging-2.3.6.RELEASE.jar
file:/workspace/logback-classic-1.2.3.jar
file:/workspace/logback-core-1.2.3.jar
file:/workspace/log4j-to-slf4j-2.13.3.jar
file:/workspace/log4j-api-2.13.3.jar
file:/workspace/jul-to-slf4j-1.7.30.jar
file:/workspace/jakarta.annotation-api-1.3.5.jar
file:/workspace/snakeyaml-1.26.jar
file:/workspace/HikariCP-3.4.5.jar
file:/workspace/slf4j-api-1.7.30.jar
file:/workspace/spring-jdbc-5.2.11.RELEASE.jar
file:/workspace/spring-beans-5.2.11.RELEASE.jar
file:/workspace/spring-tx-5.2.11.RELEASE.jar
file:/workspace/spring-boot-starter-web-2.3.6.RELEASE.jar
file:/workspace/spring-boot-starter-json-2.3.6.RELEASE.jar
file:/workspace/jackson-databind-2.11.3.jar
file:/workspace/jackson-annotations-2.11.3.jar
file:/workspace/jackson-core-2.11.3.jar
file:/workspace/jackson-datatype-jdk8-2.11.3.jar
file:/workspace/jackson-datatype-jsr310-2.11.3.jar
file:/workspace/jackson-module-parameter-names-2.11.3.jar
file:/workspace/spring-boot-starter-tomcat-2.3.6.RELEASE.jar
file:/workspace/tomcat-embed-core-9.0.39.jar
file:/workspace/jakarta.el-3.0.3.jar
file:/workspace/tomcat-embed-websocket-9.0.39.jar
file:/workspace/spring-web-5.2.11.RELEASE.jar
file:/workspace/spring-webmvc-5.2.11.RELEASE.jar
file:/workspace/spring-aop-5.2.11.RELEASE.jar
file:/workspace/spring-context-5.2.11.RELEASE.jar
file:/workspace/spring-expression-5.2.11.RELEASE.jar
file:/workspace/flyway-core-6.4.4.jar
file:/workspace/h2-1.4.200.jar
file:/workspace/spring-core-5.2.11.RELEASE.jar
file:/workspace/spring-jcl-5.2.11.RELEASE.jar

The file:/workspace/BOOT-INF/lib/*.jar entry for each jar is correct. The file:/workspace/*.jar entry is incorrect as the files does not exist. The change in Flyway 7 means that each of these non-existent jars on the classpath triggers a warning.

The unwanted entries are contributed by the class path index. It has a contains entry check that is intended to prevent this duplication. It isn't working at the moment as the index's entries are just the jar's name (for example HikariCP-3.4.5.jar) while the names that are being checked against the index are the jar's path (for example BOOT-INF/lib/HikariCP-3.4.5.jar).

Manually editing the index file to prepend each entry with BOOT-INF/lib resolves the problem.

Comment From: wilkinsona

When building with Maven at least, things worked fine up to and including Boot 2.3.0.M3 where the index had contents similar to the following:

BOOT-INF/lib/spring-boot-starter-2.3.0.M3.jar
BOOT-INF/lib/spring-boot-2.3.0.M3.jar
BOOT-INF/lib/spring-context-5.2.4.RELEASE.jar
BOOT-INF/lib/spring-aop-5.2.4.RELEASE.jar
BOOT-INF/lib/spring-beans-5.2.4.RELEASE.jar
BOOT-INF/lib/spring-expression-5.2.4.RELEASE.jar
BOOT-INF/lib/spring-boot-autoconfigure-2.3.0.M3.jar
BOOT-INF/lib/spring-boot-starter-logging-2.3.0.M3.jar
BOOT-INF/lib/logback-classic-1.2.3.jar
BOOT-INF/lib/logback-core-1.2.3.jar
BOOT-INF/lib/log4j-to-slf4j-2.13.1.jar
BOOT-INF/lib/log4j-api-2.13.1.jar
BOOT-INF/lib/jul-to-slf4j-1.7.30.jar
BOOT-INF/lib/jakarta.annotation-api-1.3.5.jar
BOOT-INF/lib/spring-core-5.2.4.RELEASE.jar
BOOT-INF/lib/spring-jcl-5.2.4.RELEASE.jar
BOOT-INF/lib/snakeyaml-1.26.jar
BOOT-INF/lib/slf4j-api-1.7.30.jar

In 2.3.0.M4, the contents then changed and lost the BOOT-INF/lib prefixes:

spring-boot-starter-2.3.0.M4.jar
spring-boot-2.3.0.M4.jar
spring-context-5.2.5.RELEASE.jar
spring-aop-5.2.5.RELEASE.jar
spring-beans-5.2.5.RELEASE.jar
spring-expression-5.2.5.RELEASE.jar
spring-boot-autoconfigure-2.3.0.M4.jar
spring-boot-starter-logging-2.3.0.M4.jar
logback-classic-1.2.3.jar
logback-core-1.2.3.jar
log4j-to-slf4j-2.13.1.jar
log4j-api-2.13.1.jar
jul-to-slf4j-1.7.30.jar
jakarta.annotation-api-1.3.5.jar
spring-core-5.2.5.RELEASE.jar
spring-jcl-5.2.5.RELEASE.jar
snakeyaml-1.26.jar
slf4j-api-1.7.30.jar

At this point, the classpath contains two entries for each jar, one that exists and one that does not and the entries for existing jars are not in the correct order.

Comment From: wilkinsona

The change in M4 was, I think, due to https://github.com/spring-projects/spring-boot/issues/20564. #20564 with superseded by #20529 and we concluded that #20564 wasn't needed but I didn't revert some changes that were made for #20564 before closing it.

@mbhave do you agree with the above? If so, I think we can revert https://github.com/spring-projects/spring-boot/commit/ad164269e9aa7d0f39362d78d946a146dac5fdee to fix this.

Comment From: mbhave

@wilkinsona Yes I agree with your analysis. Since https://github.com/spring-projects/spring-boot/commit/ad164269e9aa7d0f39362d78d946a146dac5fdee also introduced a bug that jars are not loaded in a consistent order, should we create a separate issue (that can be closed by the same commit) for that so that it appears in the release notes?

Comment From: wilkinsona

Thanks, @mbhave. I've retitled this issue to hopefully capture both parts of the problem.