Running a spring boot application inside a read-only (security enhanced) docker container should be supported.
Currently (as of version 1.5.2) this doesn't work because of org.springframework.boot.context.embedded.AbstractEmbeddedServletContainerFactory#createTempDir, which creates a temporary directory for the servlet container (underneath /tmp).
When trying to start a container with --read-only, this is in the logs:
2017-03-11 18:12:46.532 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to create tempDir. java.io.tmpdir is set to /tmp
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:137) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:536) ~[spring-context-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at myapp.Application.main(Application.java:13) [classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111-internal]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111-internal]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111-internal]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app.jar:na]
Caused by: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to create tempDir. java.io.tmpdir is set to /tmp
at org.springframework.boot.context.embedded.AbstractEmbeddedServletContainerFactory.createTempDir(AbstractEmbeddedServletContainerFactory.java:218) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.context.embedded.undertow.UndertowEmbeddedServletContainerFactory.getCanonicalDocumentRoot(UndertowEmbeddedServletContainerFactory.java:484) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.context.embedded.undertow.UndertowEmbeddedServletContainerFactory.getDocumentRootResourceManager(UndertowEmbeddedServletContainerFactory.java:467) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.context.embedded.undertow.UndertowEmbeddedServletContainerFactory.createDeploymentManager(UndertowEmbeddedServletContainerFactory.java:377) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.context.embedded.undertow.UndertowEmbeddedServletContainerFactory.getEmbeddedServletContainer(UndertowEmbeddedServletContainerFactory.java:228) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:164) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:134) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
... 16 common frames omitted
Caused by: java.io.IOException: Read-only file system
at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.8.0_111-internal]
at java.io.File.createTempFile(File.java:2024) ~[na:1.8.0_111-internal]
at java.io.File.createTempFile(File.java:2070) ~[na:1.8.0_111-internal]
at org.springframework.boot.context.embedded.AbstractEmbeddedServletContainerFactory.createTempDir(AbstractEmbeddedServletContainerFactory.java:209) ~[spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
... 22 common frames omitted
For testing purposes, I've adjusted the createTempDir method by just returning new File("/tmp") - the container started up without errors and my application worked. Tested with Tomcat and Undertow.
The servlet spec (3.0 & 3.1) says in 4.8.1:
A temporary storage directory is required for each servlet context.
But it doesn't say that the directory has to be writable. ;-)
What do you think about an additional spring boot property to set read-only mode explicitly?
Comment From: bclozel
Looks like the spirit of that feature is to:
restrict the locations that an application inside a container can write files to. By using this capability in combination with volumes, you can make sure containers only persist data where it can be managed in a known location.
In my opinion, a Java application should always run with a writable "java.io.tmpdir" location, otherwise many applications and libraries won't work anyway. In your case, if the /tmp really is read-only, any call to File.createTempFile or Path. createTempDirectory will fail. I even suspect that Servlet containers don't support this, but you'd have to check that with the Tomcat/Undertow/Jetty teams.
I think this is rather an issue with best practices about setting up read-only docker images rather than a missing configuration property in Boot.
Are you sure the Servlet containers and Java applications support running on a host without a writable tmp location? Doesn't docker recommend mounting writable volumes for such use cases?
Comment From: osiegmar
Thanks for your comments!
My application writes data only to external databases and queues and doesn't need to write anything to local (neither container nor host) disk.
Security vulnerabilities like CVE-2017-5638 are using the temp directory for downloading (and executing) malware. I know, that this specific CVE is regarding Apache Struts, but you get the idea...
Are you sure the Servlet containers and Java applications support running on a host without a writable tmp location?
As written in my initial post, I've tested my (Spring Boot based) application with read-only disk for some time (using Tomcat and Undertow) and didn't notice any shortcomings. This was only possible by disabling the functionality in org.springframework.boot.context.embedded.AbstractEmbeddedServletContainerFactory#createTempDir, of course.
Doesn't docker recommend mounting writable volumes for such use cases?
Well, docker recommends securing containers as much as possible. The questions is always, what is possible with the application in use. If Spring Boot has no option for disabling the creation of a temp directory, it is not possible to use read-only root filesystem without enabling write access to temp (either via a separated mounted volume or --tmpfs).
Comment From: wilkinsona
If you were using Tomcat, I think setting server.tomcat.basedir would be sufficient
Comment From: osiegmar
No, server.tomcat.basedir is unrelated to this topic.
Comment From: wilkinsona
I disagree. If server.tomcat.basedir isn't set, a temporary directory is created.
Comment From: osiegmar
I've set server.tomcat.basedir, but the exception message kept the same. That's why I thought, it is unrelated.
I've figured out, that there are two methods in org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory that are trying to create a temp directory - getEmbeddedServletContainer and prepareContext. Setting server.tomcat.basedir only helps for getEmbeddedServletContainer. For the prepareContext I had to create a new directory static within the working directory of my docker container. This is because of AbstractEmbeddedServletContainerFactory.getValidDocumentRoot.
I've put everything together in an example app.
So, thanks for your input! Now, there's a working workaround example. But I consider this "solution" as an ugly and fragile hack.
I'd love to see real support for read-only environments directly in Spring Boot!
Comment From: bclozel
We're cleaning out the issue tracker and closing issues that we've not seen much demand to fix. Feel free to comment with additional justifications if you feel that this one should not have been closed.
Comment From: nhoughto
Just an FYI for those who get here from googles, that if you use ByteBuddy or equivalent and bytecode rewrite the createTempDir method not to throw nothing else fails, there is no progression failure that prevents you from using this in read-only containers if you fix this one minor part of the code.
Such a minor thing really makes you wonder why it isn't just supported, since it is best practice after all.
Comment From: Webby980
@bclozel For work I'm trying to make all my k8s containers read-only - unfortunately due to the issues stated within this thread that isn't possible. The workaround of creating directories during a docker build doesn't seem like a long-term solution, so was wondering whether this could be re-opened as an issue? I've not really got any extra justifications, though, other than wanting to follow best security practices and have my containers read-only; so not sure if it's worth me re-raising?
Comment From: bclozel
Hello @Webby980 I've just noticed the reactions on my latest comment and I've carefully read this thread again.
I can understand why securing containers as much as possible is a good idea. What I don't get is why this should extend to preventing any temporary file creation. I have yet to find a comprehensive guide about running JVM apps on read-only containers and the benefits of this approach.
Creating temporary files and directories is so central in the JVM and libraries that we just can't unilaterally make a change to support that. Servers, libraries using code generation, templating libraries, HTTP multipart parsers, codecs in general: many of those are using this feature in the JVM.
Without much support from the broader ecosystem, the Spring Boot team cannot support this use case on its own. This would mean patching external libraries (Tomcat and ByteBuddy are already listed here), which is a support nightmare for all parties involved.
As you've probably seen, we're embracing the container and k8s ecosystems in our latest 2.3 release. Given the current state of this use case, our decision still stands.
Comment From: php-coder
The workaround of creating directories during a docker build doesn't seem like a long-term solution
@Webby980 There is another way (or workaround) for Kubernetes -- mount emptyDir volumes to the places where an app tries to write. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-volume-storage/ and https://kubernetes.io/docs/concepts/storage/volumes/#emptydir for details.
I know that it's a little unrelated to Spring but I just provide it for your (and other readers) information.
Comment From: bclozel
Thanks @php-coder , this looks really interesting!
Comment From: osiegmar
Hi @bclozel
thanks for getting back to this. Let me explain my thoughts a bit further.
What I don't get is why this should extend to preventing any temporary file creation.
In the first place it's not about temporary files in particular – it's about immutable container file system (as a hardening best practice).
But this could easily be solved with a read-only container file system and dockers tmpfs mount option.
On the other hand, it is about temporary files for at least two reasons: * Additional hardening (the /tmp directory has been an attack vector in many successful hacks) * Ensure that none of the developers in a team mistakenly uses the local disk for caches or other things that should be stored in a designated service (e.g. Redis or clustered file system)
Creating temporary files and directories is so central in the JVM and libraries that we just can't unilaterally make a change to support that.
No one is asking for a breaking change, I think. It's just about not eagerly creating temporary directories without a need. That might also be a non-default option.
Servers, libraries using code generation, templating libraries, HTTP multipart parsers, codecs in general: many of those are using this feature in the JVM
I'm running spring boot applications with read-only filesystems since the creation of this ticket (+3 years) without any problems. Most libraries I've been using allow to deal with read-only file systems. But of course, there is no guarantee.
Without much support from the broader ecosystem, the Spring Boot team cannot support this use case on its own. This would mean patching external libraries (Tomcat and ByteBuddy are already listed here), which is a support nightmare for all parties involved.
I think you're making this out to be a bigger deal than it really is. Of course, not in every use case it is possible. Of course, not every library will support it. Many hardening aspects cannot be done for every application. This is not a one size fits all thing. Please remember, I was just asking for not eagerly creating a temp directory. ;-)
As you've probably seen, we're embracing the container and k8s ecosystems in our latest 2.3 release. Given the current state of this use case, our decision still stands.
Which was also a declined feature request which I opened in 2016 (#6804). ;-)
Comment From: philwebb
We discussed this issue on our weekly call today and have decided to open #21519 to look into our own use of temporary files. We don't think we can guarantee that all applications can run in an immutable container, but we can review our own code to make sure we're not making things worse.