Bug report
Spring Boot Version: 3.4.2
After upgrading to Spring Boot 3.4.2, my app is crashing on boot with the following logs:
"@timestamp":"2025-01-27T10:42:49.772470089+13:00","level":"ERROR","thread_name":"main","logger_name":"o.s.b.SpringApplication","m
essage":"Application run failed","throwable_class":"ApplicationContextException","stack_trace":"java.io.IOException: **'/..data/tls.k
ey'** is neither a file nor a directory\n\tat o.s.b.a.s.FileWatcher$WatcherThread.register(FileWatcher.java:150)\n\tat o.s.b.a.ssl.Fi
leWatcher.watch(FileWatcher.java:93)\n\t... 80 common frames omitted\nWrapped by: java.io.UncheckedIOException: Failed to register
paths for watching: [/opt/tls/tls.key, /opt/tls/tls.crt]\n\tat o.s.b.a.ssl.FileWatcher.watch(FileWatcher.java:96)\n\tat o.s.b.a.s.S
slPropertiesBundleRegistrar.watchForUpdates(SslPropertiesBundleRegistrar.java:82)\n\t... 79 common frames omitted\nWrapped by: j.la
ng.IllegalStateException: Unable to watch for reload on update\n\tat o.s.b.a.s.SslPropertiesBundleRegistrar.watchForUpdates(SslProp
ertiesBundleRegistrar.java:85)\n\tat o.s.b.a.s.SslPropertiesBundleRegistrar.lambda$registerBundles$2(SslPropertiesBundleRegistrar.j
ava:70)\n\t... 78 common frames omitted\nWrapped by: j.lang.IllegalStateException: Unable to register SSL bundle 'server'
My application.yaml has the following config to mount a certificate:
spring:
ssl:
bundle:
pem:
server:
keystore:
certificate: file:${TLS_CERT_PATH:}
private-key: file:${TLS_KEY_PATH:}
reload-on-update: true
server:
ssl:
bundle: server
enabled: ${TLS_ENABLED:false}
enabled-protocols: TLSv1.3
My k8s deployment provides the environment variables:
- TLS_ENABLED=true
- TLS_CERT_PATH=/opt/tls/tls.crt
- TLS_KEY_PATH=/opt/tls/tls.key
- KEYSTORE_PATH=/opt/tls/keystore.p12
I'm not sure where /..data/tls.key comes from seeing as there's no config that provides that.
Possibly related to #43586?
Any help is appreciated!
Comment From: wilkinsona
If it's related to #43586 then I suspect that /opt/tls/tls.key is a symlink that's pointing to /..data/tls.key. If this doesn't help, please provide a complete yet minimal example that reproduces the problem.
Comment From: TazBruce
Hey @wilkinsona, thanks for your reply! I'm unsure if ../data is a path we can use. Hopefully the below can help 😄
This article may be a helpful link for context. Kubernetes manages symlinks much differently than a regular operating system.
So, when you start an inotify monitor on “user-visible files”, the default behavior of the system call is to follow the symbolic links recursively and watch the regular file at the end of the symbolic link chain (as that’s the file that will probably get updated, but not on Kubernetes).
This is where the Kubernetes AtomicWriter implementation comes into the picture: If there’s an update to the Secret/ConfigMap, kubelet will create a new timestamped directory, write files to it, update ..data symlink to the new timestamped directory (remember, it’s something you can do atomically, and finally “delete” the old timestamped directory. It’s how the files from a Secret/ConfigMap volume are always complete and consistent with one another.
To replicate this, we'd need to run this within a cluster (perhaps with kind), and have a pod that is consuming a certificate secret by mounting it as a volume with the relevant environment vars to point to it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
revisionHistoryLimit: 3
template:
spec:
containers:
- name: app
image: docker.io/library/my-app:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8443
volumeMounts:
- name: tls
readOnly: true
mountPath: /opt/tls
volumes:
- name: tls
secret:
secretName: my-cert-secret
---
apiVersion: v1
kind: Secret
metadata:
name: my-cert-secret
type: kubernetes.io/tls
data:
# values are base64 encoded
tls.crt: |
xxxxx
tls.key: |
xxxxx
To validate this further, we've had a go at writing some unit tests to verify
@Test
void rbSymlink1(@TempDir Path tempDir, @TempDir Path tempDir2) throws Exception {
Path realFile = tempDir.resolve("realFile.txt");
Path symlink = tempDir2.resolve("symlink.txt");
Path relative = symlink.getParent().relativize(realFile);
System.out.println(realFile);
System.out.println(symlink);
System.out.println(relative);
Files.createFile(realFile);
Files.createSymbolicLink(symlink, relative);
System.out.println(Files.readSymbolicLink(symlink));
WaitingCallback callback = new WaitingCallback();
this.fileWatcher.watch(Set.of(symlink), callback);
Files.writeString(realFile, "Some content");
callback.expectChanges();
}
@Test
void rbSymlink2(@TempDir Path tempDir, @TempDir Path tempDir2) throws Exception {
Path realFile = tempDir.resolve("realFile.txt");
Path symlink = tempDir2.resolve("symlink.txt");
System.out.println(realFile);
System.out.println(symlink);
Files.createFile(realFile);
Files.createSymbolicLink(symlink, realFile);
System.out.println(Files.readSymbolicLink(symlink));
WaitingCallback callback = new WaitingCallback();
this.fileWatcher.watch(Set.of(symlink), callback);
Files.writeString(realFile, "Some content");
callback.expectChanges();
}
@Test
void rbSymlink3(@TempDir Path tempDir, @TempDir Path tempDir2) throws Exception {
Path realFile = tempDir.resolve("realFile.txt");
Path symlink = tempDir2.resolve("symlink.txt");
Path symlink2 = tempDir.resolve("symlink2.txt");
Path relative = symlink.getParent().relativize(realFile);
System.out.println(realFile);
System.out.println(symlink);
System.out.println(symlink2);
System.out.println(relative);
Files.createFile(realFile);
Files.createSymbolicLink(symlink, relative);
Files.createSymbolicLink(symlink2, symlink);
System.out.print(Files.readSymbolicLink(symlink2));
System.out.print(" -> ");
System.out.println(Files.readSymbolicLink(symlink));
WaitingCallback callback = new WaitingCallback();
this.fileWatcher.watch(Set.of(symlink2), callback);
Files.writeString(realFile, "Some content");
callback.expectChanges();
}
Interestingly enough, the third test (rbSymlink3) fails with the following:
FileWatcherTests > rbSymlink3(Path, Path) FAILED java.io.UncheckedIOException: Failed to register paths for watching: [/var/folders/bq/sj25gqrx6vv81mfpxp86x09m0000gq/T/junit-15635141749719039404/symlink2.txt] at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:96) at org.springframework.boot.autoconfigure.ssl.FileWatcherTests.rbSymlink3(FileWatcherTests.java:180)
Caused by: java.io.IOException: '/Users/xxx/src/spring-boot/spring-boot-project/spring-boot-autoconfigure/../junit-15635141749719039404/realFile.txt' is neither a file nor a directory at org.springframework.boot.autoconfigure.ssl.FileWatcher$WatcherThread.register(FileWatcher.java:150) at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:93) ... 1 more
Not 100% sure I'm on the right track, but I feel like it's something to do with how java manages working directories that have symlinks which point to relative directories (note how the IO exception leads to a ../ path which is similar to what we faced in the boot crash above)
Any help is much appreciated!
Comment From: bclozel
@TazBruce Thanks for the feedback and the link to this article.
I have replicated the k8s setup and behavior with a test:
/*
* Replicating a k8s configmap folder structure like:
*
* secret.txt -> ..data/secret.txt
* ..data/ -> ..a72e81ff-f0e1-41d8-a19b-068d3d1d4e2f/
* ..a72e81ff-f0e1-41d8-a19b-068d3d1d4e2f/secret.txt
*
* After a secret update, this will look like:
*
* secret.txt -> ..data/secret.txt
* ..data/ -> ..bba2a61f-ce04-4c35-93aa-e455110d4487/
* ..bba2a61f-ce04-4c35-93aa-e455110d4487/secret.txt
*/
@Test
void shouldTriggerOnConfigMapUpdates(@TempDir Path tempDir) throws Exception {
Path configMap1 = createConfigMap(tempDir, "secret.txt");
Path configMap2 = createConfigMap(tempDir, "secret.txt");
Path data = tempDir.resolve("..data");
Files.createSymbolicLink(data, configMap1);
Path secretFile = tempDir.resolve("secret.txt");
Files.createSymbolicLink(secretFile, data.resolve("secret.txt"));
try {
WaitingCallback callback = new WaitingCallback();
this.fileWatcher.watch(Set.of(secretFile), callback);
Files.delete(data);
Files.createSymbolicLink(data, configMap2);
FileSystemUtils.deleteRecursively(configMap1);
callback.expectChanges();
}
finally{
FileSystemUtils.deleteRecursively(configMap2);
Files.delete(data);
Files.delete(secretFile);
}
}
Path createConfigMap(Path parentDir, String secretFileName) throws IOException {
Path configMapFolder = parentDir.resolve(".." + UUID.randomUUID());
Files.createDirectory(configMapFolder);
Path secret = configMapFolder.resolve(secretFileName);
Files.createFile(secret);
return configMapFolder;
}
This test is green and I'm not replicating the issue.
Note, the article you're mentioning does not refer to relative folder but to folders with names starting with "..". In your case, it seems it's trying to resolve "/..data/tls.key" which doesn't seem to exist? I think that the "rbSymlink3" test is invalid as it's trying to watch an link to a relative path that doesn't exist.
Maybe you can show the complete file structure of the container in this case? Can you think of anything wrong with my test regarding k8s's behavior?
Thanks!
Comment From: kasprzakdanielt
In my usecase, I have a letsencrypt certbot updating my files. Same problem as OP. When I have "reload-on-update" enabled it won't start. System is Debian 12 with spring-boot running inside a docker container. Container has a mounted path under "/etc/letsencrypt:/etc/letsencrypt"
``` ls -l /etc/letsencrypt/live/XXX/fullchain.pem /etc/letsencrypt/live/XXX/fullchain.pem -> ../../archive/XXX/fullchain32.pem
config:
reload-on-update: true
keystore:
certificate: file:/etc/letsencrypt/live/XXX/fullchain.pem
private-key: file:/etc/letsencrypt/live/XXX/privkey.pem
Caused by: java.io.UncheckedIOException: Failed to register paths for watching: [/etc/letsencrypt/live/XXX/privkey.pem, /etc/letsencrypt/live/XXX/fullchain.pem] 2025-01-28T09:59:38.023118852Z at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:96) 2025-01-28T09:59:38.023127740Z at org.springframework.boot.autoconfigure.ssl.SslPropertiesBundleRegistrar.watchForUpdates(SslPropertiesBundleRegistrar.java:82) 2025-01-28T09:59:38.023136635Z ... 80 common frames omitted 2025-01-28T09:59:38.023145117Z Caused by: java.io.IOException: '/../../archive/XXX/privkey32.pem' is neither a file nor a directory 2025-01-28T09:59:38.023154025Z at org.springframework.boot.autoconfigure.ssl.FileWatcher$WatcherThread.register(FileWatcher.java:150) 2025-01-28T09:59:38.023162721Z at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:93) 2025-01-28T09:59:38.023171357Z ... 81 common frames omitted
**Comment From: bclozel**
@kasprzakdanielt I'm trying to reproduce this in a test but I'm missing something.
I'm not sure where the "/../../archive/XXX/privkey32.pem" path is coming from. Can you share the result of `ls -l /etc/letsencrypt/live/XXX/privkey.pem` which seems to be the issue here?
**Comment From: kasprzakdanielt**
@bclozel ah, sorry, I copied the wrong part of a stacktrace.
ls -l /etc/letsencrypt/live/XXX/privkey.pem /etc/letsencrypt/live/XXX/privkey.pem -> ../../archive/XXX/privkey32.pem ```
Here is a more detailed file structure: https://community.letsencrypt.org/t/certbot-file-structure-need-detailed-demo/39687/2
Comment From: bclozel
Thanks @kasprzakdanielt I managed to reproduce the behavior in a test and fixed it for the next maintenance release.
Comment From: kasprzakdanielt
Glad to help. Thanks for the fix @bclozel