This workflow used to work, but now doesn't:

apiVersion: skaffold/v2alpha4
kind: Config
build:
  artifacts:
    - image: localhost:5000/apps/demo
      custom:
        buildCommand: ./mvnw spring-boot:build-image -D spring-boot.build-image.imageName=$IMAGE && docker push $IMAGE
        dependencies:
          paths:
            - pom.xml
            - src/main/resources
            - target/classes
      sync:
        manual:
          - src: "src/main/resources/**/*"
            dest: /workspace/BOOT-INF/classes
            strip: src/main/resources/
          - src: "target/classes/**/*"
            dest: /workspace/BOOT-INF/classes
            strip: target/classes/

The problem is that Skaffold still notices changes and syncs them with the running container, but Spring Boot is now running from the JarLauncher (because of a change in the default buildpack), so the changes are ignored by the app. You can see from the logs that the main thread is "main" not "restartedMain" so you know it's not going to work.

To fix it you would need a system property or something that could be set to force the "restartMain" thread to be created even though the class loader is a JarLauncher. Or alternatively, a way to build the image with a different Main-Class.

Comment From: dsyer

Suggested workarounds:

  1. Use Dockerfile instead of Maven plugin to build image:
apiVersion: skaffold/v2beta3
kind: Config
build:
  artifacts:
    - image: localhost:5000/apps/demo
      custom:
        buildCommand: docker build -t $IMAGE . && docker push $IMAGE
        dependencies:
          paths:
            - pom.xml
            - src/main/resources
            - target/classes
            - Dockerfile
      sync:
        manual:
          - src: "src/main/resources/**/*"
            dest: /workspace/app
            strip: src/main/resources/
          - src: "target/classes/**/*"
            dest: /workspace/app
            strip: target/classes/
deploy:
  kustomize:
    paths:
      - "src/k8s/demo/"

with a Dockerfile:

# syntax=docker/dockerfile:experimental
FROM openjdk:8-jdk-alpine as build
WORKDIR /workspace/app

COPY mvnw .
COPY .mvn .mvn
COPY pom.xml .
COPY src src

RUN --mount=type=cache,target=/root/.m2 ./mvnw install -DskipTests
RUN mkdir -p target/dependency && (cd target/dependency; jar -xf ../*.jar)

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG DEPENDENCY=/workspace/app/target/dependency
WORKDIR /workspace
COPY --from=build ${DEPENDENCY}/BOOT-INF/lib app/lib
COPY --from=build ${DEPENDENCY}/META-INF app/META-INF
COPY --from=build ${DEPENDENCY}/BOOT-INF/classes app
ENTRYPOINT ["java","-cp","app:app/lib/*","com.example.demo.DemoApplication"]
  1. Call org.springframework.boot.devtools.restart.Restarter.initialize(String[], boolean, RestartInitializer) with a custom RestartInitializer that doesn’t do the app class loader check. You could do that in your main method or in an ApplicationListener on ApplicationStartingEvent

Comment From: torsten-liermann

I would now like to use Spring Boot 2.3.x, with skaffold and buildpacks.

Wouldn't it be possible to make the Main-Class - and maybe also classpath extensions - configurable similar to JAVA_OPTS?

Who is developing this buildpack?

Comment From: scottfrederick

@torsten-liermann There is more information on buildpacks at https://buildpacks.io/ and https://paketo.io/.

If you have any further questions, please follow up on Stack Overflow or Gitter. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

Comment From: jorgemoralespou

We need first class support for inner-loop development in Kubernetes, and we need to make it as simple as possible.

Comment From: wilkinsona

Given the inconsistency in behaviour between launching the app using its main class and a manually created classpath, and launching the app using the launcher and it determining the classpath, we've decided to treat this one as a bug and tackle it in 2.3.x.

Comment From: wilkinsona

I'm wavering on this again. The documentation describes the current behaviour quite accurately:

Developer tools are automatically disabled when running a fully packaged application. If your application is launched from java -jar or if it is started from a special classloader, then it is considered a “production application”. If that does not apply to you (i.e. if you run your application from a container), consider excluding devtools or set the -Dspring.devtools.restart.enabled=false system property.

Given that description it feels a bit much for a maintenance release. We may need to wait till 2.4 to change the behaviour.

Comment From: dsyer

Thumbs up for 2.4

Comment From: wilkinsona

I wonder if we need to be a bit smarter than just enabling the restarter when DevTools is present, even if it looks like a production deployment. I'm imagining a property or environment variable that can be used to enable the restarter, irrespective of what's launched the application, and something that hooks into skaffold dev to set the environment variable. This is somewhat similar to skaffold debug setting JAVA_TOOL_OPTIONS to enable debugging.

Comment From: wilkinsona

@dsyer @jorgemoralespou Any thoughts on the above?

Comment From: jorgemoralespou

I think that when developer go though the hassle of skipDevtools=false in the maven pom is because they need to have devtools available, independently of where it runs. The fact that it starts by default can be considered a security issue, but the fact that it won't run in any case when in buildpacks is also a "development" issue because developers want to be able to develop in containers. I think that as a first step, adding an ENV/configuration would really help, but eventually I would love to investigate if there are other options to, on the one hand help developers while at the other keep security sane. It might not be easy at first to have a solution that works with and without Buildpacks, but obviously with buildpacks there's a higher degree of things we can do.

It would be cool if that ENV you propose, when run from a Buildpack, could add the devtools jar and free the developer from having to tune the pom.xml or create multiple profiles. Another option could be to create 2 images when the BP detects devtools, where it produces 2 set of images with the only difference of an additional layer with the devtools.jar in that additional layer, so that developers can run the my-image:version-dev while there's also the my-image:version image that would be the one used for promotion. Just a random thoughts.

In any case, I would like us to explore ways to make development of SpringBoot apps on containers on Kubernetes clusters more seamless and possible with as many tools as possible, from IDEs to CLIs.