When in the context of a Gradle multi-project with several images to be built with bootBuildImage task and with parallel flag active, random errors occurs during the docker build.

After investigation, it seems that it is because the docker daemon is accessed concurrently. It would be nice to automatically prevent the tasks of type bootBuildImage to be run in parallel.

It could be done with either: * Gradle mustRunAfter -> https://gradle.github.io/kotlin-dsl-docs/api/org.gradle.api/-task/must-run-after.html * Gradle build shared services -> https://docs.gradle.org/current/userguide/build_services.html

Don't know if it should be the default behaviour as the tasks could be customized to use different daemon (or some daemon support concurrent access?).

Example with build shared services (recent feature of Gradle and Incubating)

var dockerDaemon = project.gradle.sharedServices.registerIfAbsent("dockerDaemon", DockerDaemonService::class) {
    maxParallelUsages.set(1)
}
tasks.withType<org.springframework.boot.gradle.tasks.bundling.BootBuildImage> {
    usesService(dockerDaemon)
}

import org.gradle.api.services.BuildService
import org.gradle.api.services.BuildServiceParameters

abstract class DockerDaemonService : BuildService<BuildServiceParameters.None>

Comment From: wilkinsona

@ghilainm Can you please share some more information about the errors that occurred? We believe that the problems with parallel builds should have been fixed by https://github.com/spring-projects/spring-boot/issues/27888. What version of Spring Boot are you using?

Comment From: ghilainm

@wilkinsona I am using org.springframework.boot:spring-boot-gradle-plugin:2.6.1

Those kind of errors:

    [creator]     Adding layer 'launcher'
    [creator]     Adding layer 'config'
    [creator]     Adding layer 'process-types'
    [creator]     Adding label 'io.buildpacks.lifecycle.metadata'
    [creator]     Adding label 'io.buildpacks.build.metadata'
    [creator]     Adding label 'io.buildpacks.project.metadata'
    [creator]     Adding label 'org.springframework.boot.version'
    [creator]     Setting default process type 'web'
    [creator]     Saving docker.io/mas/mas-mobile-account-component:latest...
    [creator]     ERROR: failed to export: saving image: failed to commit cache: backing up cache: rename /launch-cache/committed /launch-cache/committed-backup: no such file or directory

I could try to disable my fix and see if can reproduce once more.

Comment From: wilkinsona

This has been discussed in the past on Gitter where @scottfrederick suggested the following:

the message is coming from the CNB builder’s lifecycle component, which Boot launches but otherwise is out of Boot’s control. if caching with concurrent builds is an issue, you can disable caching in the gradle config: https://docs.spring.io/spring-boot/docs/current/gradle-plugin/reference/html/#build-image-customization

cleanCache=true will cause the Boot gradle plugin to delete the cache volume from Docker daemon and instruct the CNB builder lifecycle not to cache build results.

Could you please give this a try, @ghilainm?

Comment From: ghilainm

@wilkinsona I have tried with what you suggest and it works, but it didn't succeed. Now, I get another issue (but of the same kind according to me).

> Docker API call to 'localhost/v1.24/volumes/pack-cache-5e0762d6bec6.build?force=1' failed with status code 409 "Conflict" or that one [creator] ERROR: failed to export: create layer file in cache: open /launch-cache/staging/sha256:889532d10afb71168fccd0e4c84ba6d442ee565523759e0bd009b81735fb628f.tar: no such file or directory

To provide more context, I have two types of task which creates docker images:

  • org.springframework.boot.gradle.tasks.bundling.BootBuildImage
  • com.bmuschko.gradle.docker.tasks.image.DockerBuildImage

I have disabled the second one to execute the test you requested to avoid any side effect :).

So to conclude for the moment, when running in //, I get random errors.

Comment From: wilkinsona

Thanks, @ghilainm. What do you mean by "running in //"?

That behaviour's surprising as it still appears to be a cache-related failure. @scottfrederick, any idea why cleanCache=true didn't switch this off?

Comment From: scottfrederick

There are two separate caches used by the CNB lifecycle (which is the source of this error) - the build cache and the launch cache. When cleanCache=true, the Spring Boot plugin will delete the build cache volume if it exists and instruct the lifecycle not to use a build cache. The cleanCache option has no effect on the launch cache and there is not an option in the lifecycle to disable the launch cache. This is consistent with the behavior of the pack buildpack CLI that the Boot plugins are modeled after. So the advice I previously gave on Gitter was wrong.

Both caches are stored in volumes provided to the Docker daemon, and separate volumes are created for each application that an image is built for. I don't know for sure, but it seems that there should not be cache contention when two images are built for different applications at the same time. There could be contention if two processes try to build an image for the same process at the same time.

@ghilainm When the error happens, can you tell if it is separate CI processes building images for the same application, or for different applications?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

Comment From: beat2

It looks like I have the same problem with the spring-boot-maven-plugin. When two builds of the same application (two branches) run concurrently on Jenkins, one of the builds gets the error:

18:17:12 [INFO] [creator] ERROR: failed to export: saving image: failed to commit cache: committing cache: rename /launch-cache/staging /launch-cache/committed: no such file or directory

I tried with "cleanCache" but then I get:

org.springframework.boot:spring-boot-maven-plugin:3.2.3:build-image-no-fork failed: Docker API call to '/var/run/docker.sock/v1.24/volumes/pack-cache-e868185afd4e.build?force=1' failed with status code 409 "Conflict"

Am I doing something wrong?

Comment From: scottfrederick

@beat2 The message rename /launch-cache/staging /launch-cache/committed: no such file or directory indicates that you are having a clash with the launch cache when images for the same app are being built in the same Docker daemon. As explained in the comment above, cleanCache will not help with this.

Since this issue was opened, we've added the ability to customize the names and types of both the build and launch caches. You'll need to configure these to make sure they are unique to each build, not just to the application.

Comment From: beat2

@scottfrederick thank you! I was able to use a generated UUID to make both unique and this works!