Spring-boot version 3.3.9

I have configured a springboot application to use SSL bundles and enabled the hot reload functionality as below:

application.properties
#ssl bundle config
spring.ssl.bundle.pem.server.reload-on-update=true
spring.ssl.bundle.pem.server.keystore.certificate=file:/secret/tls.crt
spring.ssl.bundle.pem.server.keystore.private-key=file:/secret/tls.key
spring.ssl.bundle.pem.server.truststore.certificate=file:/secret/ca.crt
server.ssl.bundle=server

Certificates are generated by certmanager and stored as kubernetes secrets which are then mounted into the application pods at the volume paths below:

volumeMounts:
  - mountPath: /secret
    name: volume-secret
    readOnly: true 
volumes:
  - name: volume-secret
    projected:
      defaultMode: 420
      sources:
      - secret:
          name: secret-tls-springboot-app

Observation:

  1. On start up, cert-manager provisions the certs in a Kubernetes Secret and they are mounted on the pod at /secret and the application starts up just fine.
  2. When the certificate is auto renewed by the cert-manager first time the springboot SSL hot reload functionality picks up the latest changes to the certs:
{"@timestamp":"2025-03-18T16:47:19.008+00:00","classname":"org.springframework.boot.web.embedded.tomcat.SslConnectorCustomizer","method":"update","file":"SslConnectorCustomizer.java","line":63,"thread":"ssl-bundle-watcher","level":"DEBUG","component":"springboot-app","message":"SSL Bundle for host _default_ has been updated, reloading SSL configuration","exception":""}
{"@timestamp":"2025-03-18T16:47:19.156+00:00","classname":"org.apache.juli.logging.DirectJDKLog","method":"log","file":"DirectJDKLog.java","line":173,"thread":"ssl-bundle-watcher","level":"INFO","component":"springboot-app","message":"Connector [https-jsse-nio-8443], TLS virtual host [_default_], certificate type [UNDEFINED] configured from keystore [/opt/dockeruser/.keystore] using alias [tomcat] with trust store [null]","exception":""}`
  1. When the certificate is auto renewed by cert-manager for a second time, the springboot hot reload functionality does not pick up the changes and application still refers to old certificates. No logs are printed and the ssl-bundle-watcher does not seem to be triggered. Question:

Why would the SSL hot reload functionality pick up the first change to the certificate files but not pick up the second one or any further changes?

Comment From: wilkinsona

This looks very similar to the situation described in https://github.com/spring-projects/spring-boot/issues/43989#issuecomment-2738260053.

Comment From: bclozel

I think I'm missing something because I cannot reproduce this in our tests.

I've updated our org.springframework.boot.autoconfigure.ssl.FileWatcherTests with a new test:

    @Test
    void shouldTriggerOnManyConfigMapUpdates(@TempDir Path tempDir) throws Exception {
        Path configMap1 = createConfigMap(tempDir, "secret.txt");
        Path configMap2 = createConfigMap(tempDir, "secret.txt");
        Path data = tempDir.resolve("..data");
        Files.createSymbolicLink(data, configMap1);
        Path secretFile = tempDir.resolve("secret.txt");
        Files.createSymbolicLink(secretFile, data.resolve("secret.txt"));
        try {
            WaitingCallback callback = new WaitingCallback();
            this.fileWatcher.watch(Set.of(secretFile), callback);
            // first update
            Files.delete(data);
            Files.createSymbolicLink(data, configMap2);
            callback.expectChanges();

            // reset callback state
            callback.reset();

            // second update
            Files.delete(data);
            Files.createSymbolicLink(data, configMap1);
            callback.expectChanges();
        }
        finally {
            FileSystemUtils.deleteRecursively(configMap1);
            FileSystemUtils.deleteRecursively(configMap2);
            Files.delete(data);
            Files.delete(secretFile);
        }
    }

    Path createConfigMap(Path parentDir, String secretFileName) throws IOException {
        Path configMapFolder = parentDir.resolve(".." + UUID.randomUUID());
        Files.createDirectory(configMapFolder);
        Path secret = configMapFolder.resolve(secretFileName);
        Files.createFile(secret);
        return configMapFolder;
    }

    private static final class WaitingCallback implements Runnable {

        private CountDownLatch latch = new CountDownLatch(1);

        volatile boolean changed = false;

        @Override
        public void run() {
            this.changed = true;
            this.latch.countDown();
        }

        void expectChanges() throws InterruptedException {
            waitForChanges(true);
            assertThat(this.changed).as("changed").isTrue();
        }

        void expectNoChanges() throws InterruptedException {
            waitForChanges(false);
            assertThat(this.changed).as("changed").isFalse();
        }

        void waitForChanges(boolean fail) throws InterruptedException {
            if (!this.latch.await(5, TimeUnit.SECONDS)) {
                if (fail) {
                    fail("Timeout while waiting for changes");
                }
            }
        }

        void reset() {
            this.latch = new CountDownLatch(1);
            this.changed = false;
        }

    }

Maybe this test is invalid and the file system changes we are simulating are not what they are in reality? Here's what the test is doing:

➜ mkdir app config1 config2
➜ touch config1/secret.txt config2/secret.txt
➜ ln -s config1 data
➜ tree
.
├── app
├── config1
│   └── secret.txt
├── config2
│   └── secret.txt
└── data -> config1

5 directories, 2 files

➜ echo "simulating config update"
simulating config update
➜ rm data
➜ ln -s config2 data
➜ tree
.
├── app
├── config1
│   └── secret.txt
├── config2
│   └── secret.txt
└── data -> config2

5 directories, 2 files

➜ echo "changes detected"
changes detected

Can you share a set of bash commands that would simulate the tree structure and runtime filesystem changes that happen?