SpringBoot Nested jar URLs cannot be split and reassembled resulting in errors with projects that use this technique (such as JobRunr)

Hi, I'm Ronald, the creator of JobRunr.

An issue was reported to us regarding the new nested jar support which broke JobRunr as it reads files from the classpath.

Inside an Uber jar, it is currently impossible to list the files within a directory using a NIO FileSystemProvider

I would expect the following to work:

var url = ((JarURLConnection)H2StorageProvider.class.getResource("migrations").openConnection()).getJarFileURL();
FileSystem fileSystem = FileSystems.newFileSystem(url.toURI(), Collections.emptyMap());
Path path = fileSystem.getPath("BOOT-INF/lib/jobrunr-6.3.3.jar!/org/jobrunr/storage/sql/common/migrations/");
boolean exists = Files.exists(path);
List<Path> list = Files.list(path).collect(Collectors.toList());
return "Root exists: " + exists + "; contents: " + list;

The FileSystem returned is of type class org.springframework.boot.loader.nio.file.NestedFileSystem.

For a reproducible scenario, please see https://github.com/rdehuyss/spring-boot-3.2-nested-jar-issue

Comment From: rdehuyss

Tagging @philwebb as this related to: - https://github.com/spring-projects/spring-boot/issues/37668 - https://github.com/spring-projects/spring-boot/commit/7ad4a9817da4b04c1a743cc52a7585dd30b25ef9

Comment From: rdehuyss

Also related:

FileSystems.getFileSystem(URI.create("jar:nested:/Users/rdehuyss/Projects/Personal/jobrunr/examples/example-java-mag/target/example-java-mag-1.0-SNAPSHOT.jar"));

throws the following exception (which I can understand from the docs as it is the new format to retrieve resources):

java.lang.IllegalArgumentException: 'path' must contain '/!'
        at org.springframework.boot.loader.net.protocol.nested.NestedLocation.parse(NestedLocation.java:98)
        at org.springframework.boot.loader.net.protocol.nested.NestedLocation.fromUri(NestedLocation.java:89)
        at org.springframework.boot.loader.nio.file.NestedFileSystemProvider.getPath(NestedFileSystemProvider.java:88)
        at java.base/java.nio.file.Path.of(Path.java:208)
        at java.base/java.nio.file.Paths.get(Paths.java:98)
        at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.uriToPath(ZipFileSystemProvider.java:76)
        at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:98)
        at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:339)
        at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:288)

However,

FileSystems.newFileSystem(URI.create("jar:nested:/Users/rdehuyss/Projects/Personal/jobrunr/examples/example-java-mag/target/example-java-mag-1.0-SNAPSHOT.jar/!BOOT-INF/lib/jobrunr-1.0.0-SNAPSHOT.jar"), Collections.emptyMap());

throws the following exception:

java.util.zip.ZipException: read CEN tables failed
        at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.initCEN(ZipFileSystem.java:1549)
        at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:174)
        at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getZipFileSystem(ZipFileSystemProvider.java:125)
        at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:106)
        at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:339)
        at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:288)

Comment From: philwebb

Thanks very much for the detailed report @rdehuyss. I think we've actually got two distinct bugs here so I've opened #38595 for the second.

I think I have a fix for the sample project provided in https://github.com/jobrunr/jobrunr/issues/884 but I'm not sure about the reproducer you attached to this issue.

As it stands, that your sample is calling H2StorageProvider.class.getResource("migrations") which returns a URL in the form jar:nested:/the.jar/!BOOT-INF/lib/jobrunr-6.3.3.jar!/org/jobrunr/storage/sql/h2/migrations.

It then gets the JAR file URL which is nested:/the.jar/!BOOT-INF/lib/jobrunr-6.3.3.jar and creates a FileSystem from it. Finally, it calls getPath("BOOT-INF/lib/jobrunr-6.3.3.jar!/org/jobrunr/storage/sql/common/migrations/").

There are a couple of problems here: 1) The filesystem created is not a ZipFileSystem 2) The path include the nested jar in it.

If I change the sample as follows, things work.

@GetMapping("/get/from/nested/jar")
public String getFromNestedJar() throws IOException, URISyntaxException {
    loadFileSystemIfNecessary();
    Path path = fileSystem.getPath("org/jobrunr/storage/sql/common/migrations/");
    boolean exists = Files.exists(path);
    List<Path> list = Files.list(path).collect(Collectors.toList());
    return "Root exists: " + exists + "; contents: " + list;
}

private static void loadFileSystemIfNecessary() throws IOException, URISyntaxException {
    if (fileSystem == null) {
        URL resource = H2StorageProvider.class.getResource("migrations");
        var jarFileUrl = ((JarURLConnection) resource.openConnection()).getJarFileURL();
        fileSystem = FileSystems.newFileSystem(new URI("jar:" + jarFileUrl), Collections.emptyMap());
    }
}

I think this makes sense because if we remove nested JARs from the picture, the original sample wouldn't work. For example, the following code:

URL url = new URL("jar:file:/the.jar!/");
JarURLConnection connection = (JarURLConnection) url.openConnection();
URL jarFileUrl = connection.getJarFileURL();
FileSystem fileSystem = FileSystems.newFileSystem(jarFileUrl.toURI(), Collections.emptyMap());

Fails when FileSystems.newFileSystem is called with:

Exception in thread "main" java.lang.IllegalArgumentException: Path component should be '/'
    at java.base/sun.nio.fs.UnixFileSystemProvider.checkUri(UnixFileSystemProvider.java:81)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newFileSystem(UnixFileSystemProvider.java:90)
    at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:339)
    at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:288)
    at org.spring.boot.nested.Test.main(Test.java:16)

In other words, we always need the jar: scheme if we want to look at the jar contents.

I hope that makes sense and the fix works OK. If it doesn't, please let me know.

Comment From: rdehuyss

Hi @philwebb - thanks for your swift intervention and input.

Just to be 100% sure, I changed the reproducer project to the above example you gave (still using Spring Boot 3.2.0). Is it possible I then encounter the issue about java.util.zip.ZipException: read CEN tables failed? I assume this will then be fixed in Spring Boot 3.2.1?

Comment From: philwebb

That correct. There's a 3.2.1-SNAPSHOT available if you want to give that a try. It's in repo.spring.io/snapshot. You can use start.spring.io to generate a project with the maven config you need.

Comment From: rdehuyss

Will do! Any idea on a release date already?

Comment From: bclozel

@rdehuyss it's scheduled on December 21st, see https://github.com/spring-projects/spring-boot/milestones and https://calendar.spring.io

Comment From: rdehuyss

I should have known that there is a release calendar 🤓.

Comment From: rdehuyss

Hi @philwebb , @wilkinsona, @bclozel : I can confirm that JobRunr works again with Spring Boot 3.2.1-SNAPSHOT.

Has anybody already mentioned you guys are awesome?! Thanks for the swift handling and feedback. I ❤️ that I do not need to change anything in JobRunr 🎉 (which I was expecting).