SpringBoot Query and ref are lost from jar: URLs that are processed by fat jar loader's handler

When a ref is used in a java.net.URL, org.springframework.boot.loader.jar.Handler.parseURL() lose the information.

Code to reproduce:

URL url = new URL("jar:file:/archive.jar!/file.txt#some-ref");
System.out.println(url);

Actual output:

jar:file:/archive.jar!/file.txt

Expected output:

jar:file:/archive.jar!/file.txt#some-ref

This bug occurs only when the application is started with spring-boot-loader. Some frameworks need reference to work properly, this information cannot be lost.

Comment From: wilkinsona

This is the same issue as was reported in https://github.com/spring-projects/spring-boot/issues/11198 but never went anywhere.

@TristanDupont Can you please provide a concrete example of the need for the URL fragment? A sample that uses a framework that needs it to work properly would be ideal.

Comment From: TristanDupontICE

Here is an simple example: https://github.com/TristanDupont/sprint_boot_url Once started just need to call http://localhost:8080/hello

The behavior is different when the application is directly run or is run by spring-boot-loader.

(~/.javacpp/cache/ should be cleared between each test)

Comment From: wilkinsona

Thanks for the sample. Having tweaked it to run on macOS, I believe I've reproduced the problem. The app works when run with mvn spring-boot:run but fails when run with java -jar. The failure is the following:

java.lang.UnsatisfiedLinkError: /Users/awilkinson/.javacpp/cache/openblas-0.3.0-1.4.2-macosx-x86_64.jar/org/bytedeco/javacpp/macosx-x86_64/libjniopenblas_nolapack.dylib: dlopen(/Users/awilkinson/.javacpp/cache/openblas-0.3.0-1.4.2-macosx-x86_64.jar/org/bytedeco/javacpp/macosx-x86_64/libjniopenblas_nolapack.dylib, 1): Library not loaded: @rpath/libopenblas_nolapack.dylib
  Referenced from: /Users/awilkinson/.javacpp/cache/openblas-0.3.0-1.4.2-macosx-x86_64.jar/org/bytedeco/javacpp/macosx-x86_64/libjniopenblas_nolapack.dylib
  Reason: image not found
    at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_151]
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_151]
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_151]
    at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_151]
    at java.lang.System.load(System.java:1086) ~[na:1.8.0_151]
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1205) ~[javacpp-1.4.2.jar!/:1.4.2]
    at org.bytedeco.javacpp.Loader.load(Loader.java:983) ~[javacpp-1.4.2.jar!/:1.4.2]
    at org.bytedeco.javacpp.Loader.load(Loader.java:882) ~[javacpp-1.4.2.jar!/:1.4.2]
    at org.bytedeco.javacpp.openblas_nolapack.<clinit>(openblas_nolapack.java:10) ~[openblas-0.3.0-1.4.2.jar!/:1.4.2]
    at java.lang.Class.forName0(Native Method) ~[na:1.8.0_151]
    at java.lang.Class.forName(Class.java:348) ~[na:1.8.0_151]
    at org.bytedeco.javacpp.Loader.load(Loader.java:941) ~[javacpp-1.4.2.jar!/:1.4.2]
    at org.bytedeco.javacpp.Loader.load(Loader.java:898) ~[javacpp-1.4.2.jar!/:1.4.2]
    at org.bytedeco.javacpp.presets.openblas_nolapack.blas_set_num_threads(openblas_nolapack.java:189) ~[openblas-0.3.0-1.4.2.jar!/:0.3.0-1.4.2]
    at sample.controller.SimpleController.healthCheck(SimpleController.java:20) ~[classes!/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

The relationship between the failure and the loss of the #whatever portion of the URL isn't clear from the error message and exception. Can you explain how you made that connection please?

Comment From: TristanDupontICE

I did not look how the library use the reference, but I looked where the information is lost.

When a new java.lang.URL is created, the constructor does all the parsing job, then a java.net.URLStreamHandler is called to finish the parsing. The reference part is properly parsed by the URL.

When we start a spring boot application with the spring-boot-loader, the org.springframework.boot.loader is registered as java.protocol.handler.pkgs and org.springframework.boot.loader.jar.Handler is used as URLStreamHandler. But this handler does not use all parsed information of the URL source:

private void setFile(URL context, String file) {
    setURL(context, JAR_PROTOCOL, null, -1, null, null, normalize(file), null, null);
}

The reference information is available in the context object but the method always set a null value.

I don't know if this behavior is voluntary or if it's mistake.

Comment From: wilkinsona

Answering my own question, the connection between the failure and the URL's ref is this line in org.bytedeco.javacpp.Loader where it creates a URL with a ref:

u = new URL(u + "#" + styles2[i]);

The ref is then used subsequently when naming the resource in the cache:

if (resourceURL.getRef() != null) {
    // ... get the URL fragment to let users rename library files ...
    String newName = resourceURL.getRef();
    // ... but create a symbolic link only if the name does not change ...
    reference = newName.equals(name);
    name = newName;
}

This is certainly an unusal usage of the URL's ref, but it's supported by the JDK's handler for jar: URLs so ours should support it too.

Comment From: wilkinsona

The problem applies to query as well. It is slightly more complicated as, to match the JDK handler's behaviour, it needs to be parsed out of the file that's created from the spec.

Comment From: CristianPi

Sorry to re open this old issue but they seems related JarURLConnection.

Comment From: wilkinsona

@CristianPi Looking at https://github.com/bytedeco/javacpp-presets/issues/1319, I can't see any connection to this issue. There's no apparent use of query or ref in the jar: URL.

Comment From: CristianPi

@CristianPi Looking at bytedeco/javacpp-presets#1319, I can't see any connection to this issue. There's no apparent use of query or ref in the jar: URL.

Yeah i don't know, do you think this is a new issue? the bytedeco pkg is bundled by spring boot, but it does not work inside the jar, with missing file errors. It works fine without bundling the jar. Maybe the JarURLConnection is connected somehow? there's a few years of development since this issue, so maybe the original problem is not really applicable.

Comment From: wilkinsona

This issue definitely isn't applicable as there's no query or ref in the URL involved.

Unless this has worked previously, I suspect that the code in javacpp-presets doesn't understand nested jar files. https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/index.html#howto.build.extract-specific-libraries-when-an-executable-jar-runs is a work around for this limitation.

If it has worked with earlier versions of Spring Boot when bundled as an executable jar, please open a new issue with a minimal, reproducible example of the problem.