When trying to configure buildpacks for the bootBuildImage target (from: https://github.com/spring-projects/spring-boot/issues/25561#issuecomment-795615788)
bootBuildImage {
buildpacks = [
'gcr.io/paketo-buildpacks/eclipse-openj9:latest',
'paketo-buildpacks/java'
]
}
will result in an error
> Task :bootBuildImage
Building image 'demo:latest'
> Pulling builder image 'docker.io/paketobuildpacks/builder:base' ..................................................
> Pulled builder image 'paketobuildpacks/builder@sha256:148282212248cd49988c0cdddff19ee7091f02b03b2b4bce52e5d68640e7b0c5'
> Pulling run image 'docker.io/paketobuildpacks/run:base-cnb' ..................................................
> Pulled run image 'paketobuildpacks/run@sha256:956341a43edf4029b29114502f7deb185527aed63e2a5ab94d86cd33abb34e56'
> Pulling buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9:latest' ..................................................
> Pulled buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9@sha256:9ec7ecf806716d1416b6f9030dea08a33cf1ed2d35dff5c2c4cab0d7f46cd4d1'
> Executing lifecycle version v0.11.3
> Using build cache volume 'pack-cache-877d4fc3db41.build'
> Running creator
[creator] ===> DETECTING
[creator] ERROR: failed to lookup buildpack.toml for buildpack 'paketo-buildpacks/eclipse-openj9@8.0.0': open /cnb/buildpacks/paketo-buildpacks_eclipse-openj9/8.0.0/buildpack.toml: permission denied
> Task :bootBuildImage FAILED
FAILURE: Build failed with an exception.
Without custom buildpacks everything works fine and the image is created.
Checking the docker image gcr.io/paketo-buildpacks/eclipse-openj9:latest it seems that the buildpack.toml is in place with read all rights.
- Spring boot 2.5
- Gradle 7.0
Comment From: scottfrederick
@bala-striva I can't reproduce the error you are getting. When I run bootBuildImage with those buildpacks it works fine, with the same SHAs for the builder, run, and buildpack images:
Building image 'docker.io/library/build-demo:0.0.1-SNAPSHOT'
> Pulling builder image 'docker.io/paketobuildpacks/builder:base' ..................................................
> Pulled builder image 'paketobuildpacks/builder@sha256:148282212248cd49988c0cdddff19ee7091f02b03b2b4bce52e5d68640e7b0c5'
> Pulling run image 'docker.io/paketobuildpacks/run:base-cnb' ..................................................
> Pulled run image 'paketobuildpacks/run@sha256:956341a43edf4029b29114502f7deb185527aed63e2a5ab94d86cd33abb34e56'
> Pulling buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9:latest' ..................................................
> Pulled buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9@sha256:9ec7ecf806716d1416b6f9030dea08a33cf1ed2d35dff5c2c4cab0d7f46cd4d1'
> Executing lifecycle version v0.11.3
> Using build cache volume 'pack-cache-4a387a19819.build'
> Running creator
[creator] ===> DETECTING
[creator] 6 of 19 buildpacks participating
[creator] paketo-buildpacks/eclipse-openj9 8.0.0
[creator] paketo-buildpacks/ca-certificates 2.2.0
[creator] paketo-buildpacks/bellsoft-liberica 8.0.0
[creator] paketo-buildpacks/executable-jar 5.0.0
[creator] paketo-buildpacks/dist-zip 4.0.0
[creator] paketo-buildpacks/spring-boot 4.3.0
[creator] ===> ANALYZING
[creator] Previous image with name "docker.io/library/build-demo:0.0.1-SNAPSHOT" not found
[creator] ===> RESTORING
[creator] ===> BUILDING
[creator]
[creator] Paketo Eclipse OpenJ9 Buildpack 8.0.0
[creator] https://github.com/paketo-buildpacks/eclipse-openj9
[creator] Build Configuration:
[creator] $BP_JVM_VERSION 8.* the Java version
[creator] Launch Configuration:
[creator] $BPL_JVM_HEAD_ROOM 0 the headroom in memory calculation
[creator] $BPL_JVM_LOADED_CLASS_COUNT 35% of classes the number of loaded classes in memory calculation
[creator] $BPL_JVM_THREAD_COUNT 250 the number of threads in memory calculation
[creator] $JAVA_TOOL_OPTIONS the JVM launch flags
[creator] OpenJ9 JRE 8.0.292: Contributing to layer
[creator] Downloading from https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u292-b10_openj9-0.26.0/OpenJDK8U-jre_x64_linux_openj9_8u292b10_openj9-0.26.0.tar.gz
[creator] Verifying checksum
[creator] Expanding to /layers/paketo-buildpacks_eclipse-openj9/jre
[creator] Adding 129 container CA certificates to JVM truststore
[creator] Writing env.launch/BPI_APPLICATION_PATH.default
[creator] Writing env.launch/BPI_JVM_CACERTS.default
[creator] Writing env.launch/BPI_JVM_CLASS_COUNT.default
[creator] Writing env.launch/BPI_JVM_EXT_DIR.default
[creator] Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
[creator] Writing env.launch/JAVA_HOME.default
[creator] Writing env.launch/MALLOC_ARENA_MAX.default
...
[creator] Saving docker.io/library/build-demo:0.0.1-SNAPSHOT...
[creator] *** Images (c14b0f2252cd):
[creator] docker.io/library/build-demo:0.0.1-SNAPSHOT
Successfully built image 'docker.io/library/build-demo:0.0.1-SNAPSHOT'
I've seen the buildpack.toml: permission denied error before, but it's always been caused by real permissions problems on the file in the buildpack image and that doesn't seem to be the case here.
Can you share a minimal build.gradle file that reproduces the problem for you as well as the OS you are running Gradle on (Windows, MacOS, Linux?) and the Docker daemon you are using (Docker Desktop or something else)?
Comment From: bala-striva
Sure, with a sample generated project with reactive web dependency I can reproduce the issue on my machine.
build.gradle
plugins {
id 'org.springframework.boot' version '2.5.0'
id 'io.spring.dependency-management' version '1.0.11.RELEASE'
id 'java'
}
group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '11'
repositories {
mavenCentral()
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-webflux'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'io.projectreactor:reactor-test'
}
test {
useJUnitPlatform()
}
bootBuildImage {
buildpacks = [
'gcr.io/paketo-buildpacks/eclipse-openj9:latest',
'paketo-buildpacks/java'
]
}
OS:
Arch Linux 5.12.6-arch1-1 #1 SMP PREEMPT Sun, 23 May 2021 00:45:50 +0000 x86_64 GNU/Linux
Docker from arch linux package repository (https://archlinux.org/packages/community/x86_64/docker/):
Client:
Version: 20.10.6
API version: 1.41
Go version: go1.16.3
Git commit: 370c28948e
Built: Mon Apr 12 14:10:41 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.6
API version: 1.41 (minimum version 1.12)
Go version: go1.16.3
Git commit: 8728dd246c
Built: Mon Apr 12 14:10:25 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.5.2
GitCommit: 36cc874494a56a253cd181a1a685b44b58a2e34a.m
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Comment From: bala-striva
Using pack it seems to work on my setup.
I don't know if this is equivalent with running it from gradle.
at 09:10:40 ❯ pack build demo --path . --buildpack gcr.io/paketo-buildpacks/eclipse-openj9 --buildpack paketo-buildpacks/java --builder paketobuildpacks/builder:base
base: Pulling from paketobuildpacks/builder
Digest: sha256:148282212248cd49988c0cdddff19ee7091f02b03b2b4bce52e5d68640e7b0c5
Status: Image is up to date for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:956341a43edf4029b29114502f7deb185527aed63e2a5ab94d86cd33abb34e56
Status: Image is up to date for paketobuildpacks/run:base-cnb
latest: Pulling from paketo-buildpacks/eclipse-openj9
Digest: sha256:9ec7ecf806716d1416b6f9030dea08a33cf1ed2d35dff5c2c4cab0d7f46cd4d1
Status: Image is up to date for gcr.io/paketo-buildpacks/eclipse-openj9:latest
===> DETECTING
8 of 19 buildpacks participating
paketo-buildpacks/eclipse-openj9 8.0.0
paketo-buildpacks/ca-certificates 2.2.0
paketo-buildpacks/bellsoft-liberica 8.0.0
paketo-buildpacks/gradle 5.2.0
paketo-buildpacks/executable-jar 5.0.0
paketo-buildpacks/apache-tomcat 5.4.0
paketo-buildpacks/dist-zip 4.0.0
paketo-buildpacks/spring-boot 4.3.0
===> ANALYZING
Previous image with name "demo" not found
===> RESTORING
===> BUILDING
Paketo Eclipse OpenJ9 Buildpack 8.0.0
https://github.com/paketo-buildpacks/eclipse-openj9
Build Configuration:
$BP_JVM_VERSION 11 the Java version
Launch Configuration:
$BPL_JVM_HEAD_ROOM 0 the headroom in memory calculation
$BPL_JVM_LOADED_CLASS_COUNT 35% of classes the number of loaded classes in memory calculation
$BPL_JVM_THREAD_COUNT 250 the number of threads in memory calculation
$JAVA_TOOL_OPTIONS the JVM launch flags
OpenJ9 JDK 11.0.11: Contributing to layer
Downloading from https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.11%2B9_openj9-0.26.0/OpenJDK11U-jdk_x64_linux_openj9_11.0.11_9_openj9-0.26.0.tar.gz
Verifying checksum
Expanding to /layers/paketo-buildpacks_eclipse-openj9/jdk
Adding 129 container CA certificates to JVM truststore
Writing env.build/JAVA_HOME.override
Writing env.build/JDK_HOME.override
OpenJ9 JRE 11.0.11: Contributing to layer
Downloading from https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.11%2B9_openj9-0.26.0/OpenJDK11U-jre_x64_linux_openj9_11.0.11_9_openj9-0.26.0.tar.gz
Verifying checksum
Expanding to /layers/paketo-buildpacks_eclipse-openj9/jre
Adding 129 container CA certificates to JVM truststore
Writing env.launch/BPI_APPLICATION_PATH.default
Writing env.launch/BPI_JVM_CACERTS.default
Writing env.launch/BPI_JVM_CLASS_COUNT.default
Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
Writing env.launch/JAVA_HOME.default
Writing env.launch/MALLOC_ARENA_MAX.default
Launch Helper: Contributing to layer
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/active-processor-count
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/java-opts
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/link-local-dns
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/memory-calculator
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/openssl-certificate-loader
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/security-providers-configurer
Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/security-providers-classpath-9
JVMKill Agent 1.16.0: Contributing to layer
Downloading from https://github.com/cloudfoundry/jvmkill/releases/download/v1.16.0.RELEASE/jvmkill-1.16.0-RELEASE.so
Verifying checksum
Copying to /layers/paketo-buildpacks_eclipse-openj9/jvmkill
Writing env.launch/JAVA_TOOL_OPTIONS.append
Writing env.launch/JAVA_TOOL_OPTIONS.delim
Java Security Properties: Contributing to layer
Writing env.launch/JAVA_SECURITY_PROPERTIES.default
Writing env.launch/JAVA_TOOL_OPTIONS.append
Writing env.launch/JAVA_TOOL_OPTIONS.delim
Paketo CA Certificates Buildpack 2.2.0
https://github.com/paketo-buildpacks/ca-certificates
Launch Helper: Contributing to layer
Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Paketo BellSoft Liberica Buildpack 8.0.0
https://github.com/paketo-buildpacks/bellsoft-liberica
Build Configuration:
$BP_JVM_VERSION 11 the Java version
Launch Configuration:
$BPL_JVM_HEAD_ROOM 0 the headroom in memory calculation
$BPL_JVM_LOADED_CLASS_COUNT 35% of classes the number of loaded classes in memory calculation
$BPL_JVM_THREAD_COUNT 250 the number of threads in memory calculation
$JAVA_TOOL_OPTIONS the JVM launch flags
Paketo Gradle Buildpack 5.2.0
https://github.com/paketo-buildpacks/gradle
Build Configuration:
$BP_GRADLE_BUILD_ARGUMENTS --no-daemon assemble the arguments to pass to Gradle
$BP_GRADLE_BUILT_ARTIFACT build/libs/*.[jw]ar the built application artifact explicitly. Supersedes $BP_GRADLE_BUILT_MODULE
$BP_GRADLE_BUILT_MODULE the module to find application artifact in
Creating cache directory /home/cnb/.gradle
Compiled Application: Contributing to layer
Executing gradlew --no-daemon assemble
Downloading https://services.gradle.org/distributions/gradle-7.0.2-bin.zip
..........10%...........20%...........30%..........40%...........50%...........60%..........70%...........80%...........90%..........100%
Welcome to Gradle 7.0.2!
Here are the highlights of this release:
- File system watching enabled by default
- Support for running with and building Java 16 projects
- Native support for Apple Silicon processors
- Dependency catalog feature preview
For more details see https://docs.gradle.org/7.0.2/release-notes.html
To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.0.2/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build
> Task :compileJava
> Task :processResources
> Task :classes
> Task :bootJarMainClassName
> Task :bootJar
> Task :jar
> Task :assemble
BUILD SUCCESSFUL in 1m 7s
5 actionable tasks: 5 executed
Removing source code
Paketo Executable JAR Buildpack 5.0.0
https://github.com/paketo-buildpacks/executable-jar
Class Path: Contributing to layer
Writing env/CLASSPATH.delim
Writing env/CLASSPATH.prepend
Process types:
executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
task: java org.springframework.boot.loader.JarLauncher (direct)
web: java org.springframework.boot.loader.JarLauncher (direct)
Paketo Spring Boot Buildpack 4.3.0
https://github.com/paketo-buildpacks/spring-boot
Creating slices from layers index
dependencies
spring-boot-loader
snapshot-dependencies
application
Launch Helper: Contributing to layer
Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
Spring Cloud Bindings 1.7.1: Contributing to layer
Downloading from https://repo.spring.io/release/org/springframework/cloud/spring-cloud-bindings/1.7.1/spring-cloud-bindings-1.7.1.jar
Verifying checksum
Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
Web Application Type: Contributing to layer
Reactive web application detected
Writing env.launch/BPL_JVM_THREAD_COUNT.default
4 application slices
Image labels:
org.springframework.boot.version
===> EXPORTING
Adding layer 'paketo-buildpacks/eclipse-openj9:helper'
Adding layer 'paketo-buildpacks/eclipse-openj9:java-security-properties'
Adding layer 'paketo-buildpacks/eclipse-openj9:jre'
Adding layer 'paketo-buildpacks/eclipse-openj9:jvmkill'
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'paketo-buildpacks/executable-jar:classpath'
Adding layer 'paketo-buildpacks/spring-boot:helper'
Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding 5/5 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving demo...
*** Images (f6158dd851da):
demo
Adding cache layer 'paketo-buildpacks/eclipse-openj9:jdk'
Adding cache layer 'paketo-buildpacks/gradle:application'
Adding cache layer 'paketo-buildpacks/gradle:cache'
Successfully built image demo
Comment From: dmikusa
@bala-striva Your pack command is slightly different. It's doing a source build. When you run with Gradle, it is using what was built locally. Can you try running your pack command again and setting -p to point to the JAR/WAR that you built?
Also, add -v to the pack command for verbose output & -e BP_DEBUG=true as well. The latter will turn up the logging output from the buildpacks. It'll be a lot of output but if you can capture it when the problem occurs it could provide some clues. Thanks
Comment From: bala-striva
@dmikusa-pivotal
Running pack build demo --path ./build/libs/demo-0.0.1-SNAPSHOT.jar --buildpack gcr.io/paketo-buildpacks/eclipse-openj9 --buildpack paketo-buildpacks/java --builder paketobuildpacks/builder:base
is also working and produces the docker image.
Adding environment to build.gradle
bootBuildImage {
buildpacks = [
'gcr.io/paketo-buildpacks/eclipse-openj9:latest',
'paketo-buildpacks/java'
]
environment = ['BP_DEBUG': 'true']
}
did not produce any new log lines compared to whats added to the issue
Comment From: scottfrederick
@bala-striva Thank you for the additional information and testing, it's very helpful. The only thing that stands out to me so far is the OS you are using. We test extensively on MacOS and Ubuntu, it's possible there is something slightly different about Arch Linux that is causing your problem. I'll try to get something set up to test that combination of OS, Java, and Gradle myself.
Comment From: scottfrederick
I can reproduce this problem by testing in an Arch Linux container. I'll investigate further to find the root cause of why this works differently on Arch Linux than it does on other Linux distros.
Comment From: scottfrederick
I've confirmed that this is a bug in the Spring Boot plugins. The same failure will happen with directory and tar-gzipped buildpacks. Fixes are in progress.
Comment From: scottfrederick
@bala-striva The fix for this issue should be in a Spring Boot 2.5.1-SNAPSHOT build soon. I've tested and verified the fix on Arch Linux in a container environment, but if you get a chance to test it on your machine before the 2.5.1 release that would be very helpful.
Comment From: bala-striva
@scottfrederick with the snapshot build i was able to build the docker image with custom buildpacks.
Thanks for the detailed feedbacks!