SpringBoot Image builds with podman fail when image buildpacks are configured

Following the documentation on how to configure podman for spring boot I ended up using the following configuration in gradle:

tasks.withType<org.springframework.boot.gradle.tasks.bundling.BootBuildImage> {
    environment.put("BP_NATIVE_IMAGE", "false")
    buildpacks.set(listOf(
        "gcr.io/paketo-buildpacks/ca-certificates:3.5.1",
        "gcr.io/paketo-buildpacks/bellsoft-liberica:9.10.0",
        "gcr.io/paketo-buildpacks/syft:1.23.0",
        "gcr.io/paketo-buildpacks/executable-jar:6.5.0",
        "gcr.io/paketo-buildpacks/spring-boot:5.20.0"
    ))
    val dockerHost = System.getenv("DOCKER_HOST")

    if (dockerHost != null) {
        docker {
            host.set(dockerHost)
            bindHostToBuilder.set(true)
        }
    }
}

This first appears to work, spring boot successfully communicates with the podman daemon and then pulls the buildpacks. However, when attempting to start the build the following error occurs:

> Task :bootBuildImage
Building image 'my.company/software:fdc1ac23'
 > Pulling builder image 'docker.io/paketobuildpacks/builder:base' ..................................................
 > Pulled builder image 'docker.io/paketobuildpacks/builder@sha256:11dcec699e2349838762e245c1c961d9e2c6e10d3e3617ec1e7bba19929caa11'
 > Pulling run image 'docker.io/paketobuildpacks/run:base-cnb' ..................................................
 > Pulled run image 'docker.io/paketobuildpacks/run@sha256:b9cbbd4e962acac787cc2bdba8b9c97996675ef57155418c0c5c9cb088261434'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/ca-certificates:3.5.1' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/ca-certificates@sha256:ad50eae8fa996f0bb7c85f48135e81121fb1a5f8c9e98cf57703a623e07bd479'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/bellsoft-liberica:9.10.0' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/bellsoft-liberica@sha256:fef2cc126dfc0dd78398aa7131910d7c567e6bc799c74a2d93f577af85233cf4'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/syft:1.23.0' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/syft@sha256:7c77ba8a7dbadc613fba70063ce43b3eeb1b7782424a01bdc60de634075f7d0a'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/executable-jar:6.5.0' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/executable-jar@sha256:971568ae093406911d36fb07c35f589ee69b6c7a2d3f6b90e9e4e924f05baa13'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/spring-boot:5.20.0' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/spring-boot@sha256:0b104efc035bb6f92b38721a237a0d29e9b38fa36a02c2ffd215b2f565654aa0'
 > Executing lifecycle version v0.16.0
 > Using build cache volume 'pack-cache-068ca50ea7f6.build'
 > Running creator
    [creator]     ===> ANALYZING
    [creator]     Previous image with name "my.company/software:fdc1ac23" not found
    [creator]     ===> DETECTING
    [creator]     ERROR: failed to initialize detector: open /cnb/buildpacks/paketo-buildpacks_bellsoft-liberica/9.10.0/buildpack.toml: no such file or directory
> Task :bootBuildImage FAILED
FAILURE: Build failed with an exception.

This appears to be related to using podman, which according to the documentation is supported. Switching back to the official docker daemon is sadly not an option.

Comment From: scottfrederick

@Janrupf Thanks for the report. I can reproduce this error when using the configuration that you've shown above. If I remove the buildpacks.set(...) configuration from your sample, the image builds successfully for me. Would you mind trying to remove the custom buildpack config and see if you get the same results? That would help by confirming that we're looking at the right problem.

Comment From: Janrupf

Yes, removing the configuration works. Interestingly, the following configuration also works (I only changed the version numbers):

buildpacks.set(listOf(
    "gcr.io/paketo-buildpacks/ca-certificates:3.5.1",
    "gcr.io/paketo-buildpacks/bellsoft-liberica:9.10.3",
    "gcr.io/paketo-buildpacks/syft:1.24.2",
    "gcr.io/paketo-buildpacks/executable-jar:6.6.0",
    "gcr.io/paketo-buildpacks/spring-boot:5.22.1"
))

I additionally noticed that now my builds succeed, the image can apparently not be found whatsoever after building (the build succeeds, but the image does not exist). However, this probably is a separate issue (with podman?) or even some problem with my scripts.

Comment From: scottfrederick

Thanks for following up.

Interestingly, the following configuration also works (I only changed the version numbers)

If you use the pack CLI to inspect the builder image with pack inspect-builder paketobuildpacks/builder:base, you'll see the versions of all the buildpacks that are bundled in the builder. It appears that the buildpack versions you specified in your second configuration match the versions bundled in the most recent published builder.

When the Spring Boot plugin gets a list of buildpacks in its configuration, it will download each buildpack, compare it against buildpacks in the builder, and throw away any duplicate buildpack contents (see #31233 for the reason). So in your second example the buildpacks you specified are being used in the given order, but the buildpack contents are not being used.

I suspect something is going wrong when downloaded buildpack contents are added to the base builder contents. I'll dig deeper and see what I can find out.