Spring Boot 3.2.4 / Maven 3.9.6
When building the native image through mvn -e -Pnative spring-boot:build-image command, I get:
[...]
[INFO] --- spring-boot:3.2.4:build-image (default-cli) @
[...]
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-tiny:latest' 23%
[...]
[INFO] > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-tiny:latest' 100%
[INFO] > Pulled builder image 'paketobuildpacks/builder-jammy-tiny@sha256:f0dfd3e26cc947342dc24e8792a53edf9c42366d152b2a28f2a3d4e8c0ec70b8'
[INFO] > Pulling run image 'docker.io/paketobuildpacks/run-jammy-tiny:latest' 100%
[INFO] > Pulled run image 'paketobuildpacks/run-jammy-tiny@sha256:683d960443ba88d5eb452c62d5af70741c6ea879f37f9680fc207494d45d8980'
[INFO] > Executing lifecycle version v0.19.1
[INFO] > Using build cache volume 'pack-cache-9a30410bd6c3.build'
[INFO]
[INFO] > Running creator
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[...]
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:3.2.4:build-image (default-cli) on project validation: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:3.2.4:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 400 "Bad Request"
[...]
Caused by: org.springframework.boot.buildpack.platform.docker.transport.DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 400 "Bad Request"
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute (HttpClientTransport.java:155)
[...]
Please note that the paketobuildpacks builder image is pulled correctly, so I assume communication with docker daemon works correctly. Additionally, by checking the docker daemon (debug) logs, everything looks fine. I can even see the failing call:
DEBU[2024-04-02T14:15:54.522389260Z] Calling POST /v1.24/containers/create
DEBU[2024-04-02T14:15:54.522598897Z] form data: {"Cmd":["/cnb/lifecycle/creator","-log-level","debug","-app","/workspace","-platform","/platform","-run-image","docker.io/paketobuildpacks/run-jammy-tiny:latest","-layers","/layers","-cache-dir","/cache","-launch-cache","/launch-cache","-daemon","nexus-dan-docker-snapshot-http:30501/repository/docker/validation:latest"],"Env":["CNB_PLATFORM_API=0.12"],"HostConfig":{"Binds":["/var/run/docker.sock:/var/run/docker.sock","pack-layers-bbkagoontm:/layers","pack-app-vsyvictvgp:/workspace","pack-cache-9a30410bd6c3.build:/cache","pack-cache-9a30410bd6c3.launch:/launch-cache","/var/run/docker.sock:/var/run/docker.sock"],"SecurityOpt":["label=disable"]},"Image":"pack.local/builder/wmggxdnfdl:latest","Labels":{"author":"spring-boot"},"User":"root"}
DEBU[2024-04-02T14:15:54.572722018Z] Probing all drivers for volume with name: pack-layers-bbkagoontm
DEBU[2024-04-02T14:15:54.572811462Z] Registering new volume reference: driver "local", name "pack-layers-bbkagoontm"
DEBU[2024-04-02T14:15:54.573609321Z] Probing all drivers for volume with name: pack-app-vsyvictvgp
DEBU[2024-04-02T14:15:54.573675761Z] Registering new volume reference: driver "local", name "pack-app-vsyvictvgp"
DEBU[2024-04-02T14:15:54.574400731Z] Probing all drivers for volume with name: pack-cache-9a30410bd6c3.build
DEBU[2024-04-02T14:15:54.574451972Z] Registering new volume reference: driver "local", name "pack-cache-9a30410bd6c3.build"
DEBU[2024-04-02T14:15:54.575253795Z] Probing all drivers for volume with name: pack-cache-9a30410bd6c3.launch
DEBU[2024-04-02T14:15:54.575299081Z] Registering new volume reference: driver "local", name "pack-cache-9a30410bd6c3.launch"
DEBU[2024-04-02T14:15:54.586516529Z] Calling DELETE /v1.24/volumes/pack-layers-bbkagoontm?force=1
DEBU[2024-04-02T14:15:54.586584279Z] Removing volume reference: driver local, name pack-layers-bbkagoontm
DEBU[2024-04-02T14:15:54.587900349Z] Calling DELETE /v1.24/volumes/pack-app-vsyvictvgp?force=1
DEBU[2024-04-02T14:15:54.587947908Z] Removing volume reference: driver local, name pack-app-vsyvictvgp
DEBU[2024-04-02T14:15:54.589405293Z] Calling DELETE /v1.24/images/pack.local/builder/wmggxdnfdl:latest?force=1
Here is my setup:
I'm running the maven build on a Jenkins agent in a k8s container, docker daemon runs on another container (docker dind) inside the same Jenkins agent POD:
containers:
- name: maven
image: maven:3.9.6-eclipse-temurin-21
tty: true
command: [ "sleep" ]
args: [ "infinity" ]
...
- name: docker-daemon
image: docker:dind
tty: true
securityContext:
privileged: true
command: [ "dockerd", "--host", "tcp://127.0.0.1:2375", "--debug",...
Maven plugins setup:
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<configuration>
<debug>true</debug>
<verbose>true</verbose>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<image>
<publish>true</publish>
<name>${docker.repository.internal}/repository/docker/${project.artifactId}</name>
<verboseLogging>true</verboseLogging>
<pullPolicy>IF_NOT_PRESENT</pullPolicy>
<env>
<BP_DEBUG>true</BP_DEBUG>
</env>
<bindings>/var/run/docker.sock:/var/run/docker.sock</bindings>
</image>
<docker>
<host>tcp://localhost:2375</host>
<tlsVerify>false</tlsVerify>
<publishRegistry>
<username>***</username>
<password>***</password>
<url>${docker.repository.internal}</url>
</publishRegistry>
</docker>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
Comment From: scottfrederick
Thanks for the report and all the detail. It's hard to tell what the Docker daemon doesn't like about the call to /v1.24/containers/create. Can you get more detail from the daemon logs when the error occurs?
One thing that I'm not sure about in your configuration is the <bindings>/var/run/docker.sock:/var/run/docker.sock</bindings>. Typically that would require a valid path /var/run/docker.sock in the local environment so that it could bind the same path in the container, but since you're setting <host>tcp://localhost:2375</host> I assume you can't use /var/run/docker.sock to talk to the daemon locally. If that binding source path doesn't exist locally that might cause the daemon to throw an error.
If the tcp:// address is also available to containers with dind, you might try setting <bindHostToBuilder>true</bindHostToBuilder> in the <docker></docker> section instead. Alternatively, you can set a DOCKER_HOST environment variable, which will be used by the local Spring Boot plugin code and by the CNB lifecycle code running in the container.
Comment From: danparisi
Hi @scottfrederick, thanks for your quick reply. I did many other tests, I'm listing them below. Keep in mind that in all of them the DOCKER_HOST environment property is available:
- name: DOCKER_HOST
value: "tcp://localhost:2375"
Test 1:
1. No <host> tag
2. No <bindings> tag
3. No <bindHostToBuilder> tag
Result:
[INFO] [creator] ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: connection refused
[...]
Caused by: org.springframework.boot.buildpack.platform.build.BuilderException: Builder lifecycle 'creator' failed with status code 1
Test 2:
1. No <host> tag
2. No <bindings> tag
3. Added <bindHostToBuilder>true</bindHostToBuilder> tag
Result:
ERROR: failed to initialize analyzer: getting previous image: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
Test 3:
1. No <bindings> tag
2. No <bindHostToBuilder> tag
3. Added <host>tcp://localhost:2375</host> tag
Result:
ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: connection refused
Test 4:
1. No <host> tag
2. No <bindHostToBuilder> tag
3. Added <bindings>/var/run/docker.sock:/var/run/docker.sock</bindings> tag
Result:
DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 400 "Bad Request"
I also tried to downgrade docker daemon version and increase CPU and memory limits for the containers running the build and the docker daemon. But same result. Unluckily, I cannot find any other docker logs other the ones I listed above, it just returns 400 it seems. no debug / error log at all.
Please note that for all test cases the image is correctly pulled by the docker daemon, so communication fully work in that case (DOCKER_HOST env var is enough I assume), but the creator always fails for the reasons you can see above. Not sure if it can help. Any idea?
Comment From: scottfrederick
Unfortunately this is difficult to diagnose without access to a running environment.
Please note that for all test cases the image is correctly pulled by the docker daemon, so communication fully work in that case
This means that the Spring Boot code running locally on the host machine can talk to the Docker daemon. This is usually pretty easy to get running, as you can test local docker connectivity with the docker CLI and other tools.
but the creator always fails...
This is the hard part, as it is requires a container running in the Docker daemon to have access to the Docker host somehow (via a socket or a remote host address), and this is entirely dependent on how things like docker-in-docker are set up by a CI platform. Users have had had other issues with environments like this.
Any idea?
Logs that show what's going on when the processes running inside the container try to access the Docker host would be helpful. Otherwise, you can see if there's a way to shell into a CI container and try docker commands to see what works (for example, what sockets are available, what permissions are on the socket, if a remote host address works).
Comment From: danparisi
In the end I decided to go for another direction in order to let the pipeline works, I'll explain it below. For anyone who got the same issue, It's worth to mention that I didn't test the dind-rootless image in the setup you see above, maybe it would have fixed the issue, or maybe not: I don't know.
What I did in the end is to rely on the host machine docker daemon instead of starting up a new docker container in the Jenkins agent pod. To let it work, I just need to mount the docker socket:
apiVersion: v1
kind: Pod
spec:
volumes:
...
- hostPath:
type: Socket
path: /run/docker.sock
name: docker-sock
- hostPath:
type: Directory
path: /var/lib/docker
name: docker-lib
containers:
...
- name: maven
image: maven:3.9.6-eclipse-temurin-21
tty: true
command: [ "sleep" ]
args: [ "infinity" ]
volumeMounts:
...
- mountPath: /var/run/docker.sock
name: docker-sock
readOnly: false
- mountPath: /var/lib/docker
name: docker-lib
readOnly: false
I also got rid of DOCKER_HOST env variable and of docker host, url and bindings tags in the spring-boot-maven-plugin configuration. Everything works fine now.
Comment From: scottfrederick
@danparisi Thanks for letting us know that you found a way to get this working.