Hi,
I wanted to take a look at improving build times of BootBuildImageIntegrationTests, but noticed that 20% of them randomly fail on my local machine with messages like the following:
org.gradle.testkit.runner.UnexpectedBuildFailure: Unexpected build execution failure in /var/folders/zl/7zkcchrx0xb2t32qxx5wgh240000gp/T/gradle-4546028020545539739 with arguments [-PbootVersion=TEST-SNAPSHOT, --stacktrace, bootBuildImage, --pullPolicy=IF_NOT_PRESENT, --warning-mode, all]
Output:
> Task :compileJava
> Task :processResources
> Task :classes
> Task :bootWarMainClassName
> Task :bootWar
> Task :bootBuildImage
Building image 'docker.io/library/gradle-4546028020545539739:latest'
> Executing lifecycle version v0.10.2
> Using build cache volume 'pack-cache-6509a184f4ea.build'
> Running creator
[creator] ===> DETECTING
[creator] 5 of 18 buildpacks participating
[creator] paketo-buildpacks/ca-certificates 2.1.0
[creator] paketo-buildpacks/bellsoft-liberica 7.1.0
[creator] paketo-buildpacks/executable-jar 5.0.0
[creator] paketo-buildpacks/dist-zip 4.0.0
[creator] paketo-buildpacks/spring-boot 4.1.0
[creator] ===> ANALYZING
[creator] Previous image with name "docker.io/library/gradle-4546028020545539739:latest" not found
[creator] ===> RESTORING
[creator] ===> BUILDING
[creator]
[creator] Paketo CA Certificates Buildpack 2.1.0
[creator] https://github.com/paketo-buildpacks/ca-certificates
[creator] Launch Helper: Contributing to layer
[creator] Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[creator]
[creator] Paketo BellSoft Liberica Buildpack 7.1.0
[creator] https://github.com/paketo-buildpacks/bellsoft-liberica
[creator] Build Configuration:
[creator] $BP_JVM_VERSION 11.* the Java version
[creator] Launch Configuration:
[creator] $BPL_JVM_HEAD_ROOM 0 the headroom in memory calculation
[creator] $BPL_JVM_LOADED_CLASS_COUNT 35% of classes the number of loaded classes in memory calculation
[creator] $BPL_JVM_THREAD_COUNT 250 the number of threads in memory calculation
[creator] $JAVA_TOOL_OPTIONS the JVM launch flags
[creator] BellSoft Liberica JRE 11.0.10: Contributing to layer
[creator] Downloading from https://github.com/bell-sw/Liberica/releases/download/11.0.10+9/bellsoft-jre11.0.10+9-linux-amd64.tar.gz
[creator] Verifying checksum
[creator] Expanding to /layers/paketo-buildpacks_bellsoft-liberica/jre
[creator] Adding 129 container CA certificates to JVM truststore
[creator] Writing env.launch/BPI_APPLICATION_PATH.default
[creator] Writing env.launch/BPI_JVM_CACERTS.default
[creator] Writing env.launch/BPI_JVM_CLASS_COUNT.default
[creator] Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
[creator] Writing env.launch/JAVA_HOME.default
[creator] Writing env.launch/MALLOC_ARENA_MAX.default
[creator] Launch Helper: Contributing to layer
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/active-processor-count
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/java-opts
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/link-local-dns
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-configurer
[creator] Creating /layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/security-providers-classpath-9
[creator] JVMKill Agent 1.16.0: Contributing to layer
[creator] Downloading from https://github.com/cloudfoundry/jvmkill/releases/download/v1.16.0.RELEASE/jvmkill-1.16.0-RELEASE.so
[creator] Verifying checksum
[creator] Copying to /layers/paketo-buildpacks_bellsoft-liberica/jvmkill
[creator] Writing env.launch/JAVA_TOOL_OPTIONS.append
[creator] Writing env.launch/JAVA_TOOL_OPTIONS.delim
[creator] Java Security Properties: Contributing to layer
[creator] Writing env.launch/JAVA_SECURITY_PROPERTIES.default
[creator] Writing env.launch/JAVA_TOOL_OPTIONS.append
[creator] Writing env.launch/JAVA_TOOL_OPTIONS.delim
[creator]
[creator] Paketo Executable JAR Buildpack 5.0.0
[creator] https://github.com/paketo-buildpacks/executable-jar
[creator] Class Path: Contributing to layer
[creator] Writing env/CLASSPATH.delim
[creator] Writing env/CLASSPATH.prepend
[creator] Process types:
[creator] executable-jar: java org.springframework.boot.loader.WarLauncher (direct)
[creator] task: java org.springframework.boot.loader.WarLauncher (direct)
[creator] web: java org.springframework.boot.loader.WarLauncher (direct)
[creator]
[creator] Paketo Spring Boot Buildpack 4.1.0
[creator] https://github.com/paketo-buildpacks/spring-boot
[creator] Creating slices from layers index
[creator] dependencies
[creator] spring-boot-loader
[creator] snapshot-dependencies
[creator] application
[creator] Launch Helper: Contributing to layer
[creator] Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
[creator] Spring Cloud Bindings 1.7.0: Contributing to layer
[creator] Downloading from https://repo.spring.io/release/org/springframework/cloud/spring-cloud-bindings/1.7.0/spring-cloud-bindings-1.7.0.jar
[creator] Verifying checksum
[creator] Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
[creator] Web Application Type: Contributing to layer
[creator] Non-web application detected
[creator] Writing env.launch/BPL_JVM_THREAD_COUNT.default
[creator] 4 application slices
[creator] Image labels:
[creator] org.springframework.boot.version
[creator] ===> EXPORTING
[creator] Adding layer 'paketo-buildpacks/ca-certificates:helper'
[creator] ERROR: failed to export: caching layer (sha256:84bf98cca9651949da8909a7f9cdefaa71648359ef97fb2de93af40019bf948e): write /launch-cache/staging/sha256:84bf98cca9651949da8909a7f9cdefaa71648359ef97fb2de93af40019bf948e.tar: no space left on device
> Task :bootBuildImage FAILED
I wonder if we can improve that somewhat or how I need to tweak my Docker setup (naively giving it more space, didn't fix things)
As my original intention was to improve the time of those tests (around 30 minutes on my machine), I'd appreciate any thoughts on that, too.
Cheers, Christoph
Comment From: wilkinsona
Judging by no space left on device, I think you need to give Docker some more disk space. Could you please give that a go and let us know if things stabilise?
Comment From: dreis2211
@wilkinsona Ha - just edited the text with that. Did that already, but maybe not enough.
Comment From: wilkinsona
FWIW, my Docker install's got 128GB of disk. I had a clear out recently so it's currently only using 11.9GB. I'm sure it doesn't need over 100GB of head room, but it might be worth giving it even more space. How much free space does your install have?
Comment From: dreis2211
It had 10GB left, but I just gave it another 20GB. I'll report back in 30 minutes
Comment From: dreis2211
Yeah, 20GB more worked. Feel free to close the ticket or repurpose to a documentation ticket
Comment From: scottfrederick
I wanted to take a look at improving build times of
BootBuildImageIntegrationTestsAs my original intention was to improve the time of those tests (around 30 minutes on my machine), I'd appreciate any thoughts on that, too.
@dreis2211 I've been working on some new integration tests to verify the contracts between the Spring Boot image-building plugins and the Paketo buildpacks (for example the features documented by Paketo as enabled by Spring Boot). It's a work-in-progress right now on a branch here. The idea is that these tests would be run from different triggers than the current builds that run on every commit, such as triggering by new Paketo builder images in Docker Hub and Boot snapshots being published.
As part of this, we've started discussing changing the approach used by the existing Gradle BootBuildImageIntegrationTests and Maven BuildImageTests, which do take a long time to run. We might change the tests run on each commit to stub out a layer underneath the plugins instead of invoking the full Docker and CNB Builder paths, and leave the full-path testing to the new integration tests. We can share more information once we have time to discuss as a team again.
Comment From: scottfrederick
Created #25824 for further feedback.
Comment From: dreis2211
@scottfrederick I actually thought about extracting that from the Gradle plugin tests, but in a first try I ended up with quite some copy-paste as well. I'm somewhat glad that you ended up with that, too ;-)
My initial thought before that was to come up with a new annotation @EnabledOnCI that is only running things when on CI and not put the burden on local machines. Maybe that's a pragmatic step in between.
In general, I'm curious what you can come up with. Thanks for sharing your thoughts. :)
Comment From: scottfrederick
We can add tips for Docker resource configuration to the Working with the Code section of CONTRIBUTING.adoc.
Comment From: scottfrederick
We'll add the Docker configuration tips to the Working with the Code wiki page.
Comment From: scottfrederick
Documentation added to https://github.com/spring-projects/spring-boot/wiki/Working-with-the-Code#tests-requiring-docker.
I cleaned all containers, images, and volumes from my local Docker and ran a full build with tests. Docker reported using just under 20GB of the disk image space, so hopefully the documented recommendation holds up for others.