Currently, we are using Spring boot provided bootBuildImage gradle task to create an OCI compliant image to be deployed on our cloud foundry based cloud platform. This worked quiet well until the 29th of July 2022. The issue that we got now is happening during deployment of the image in cloud foundry. The error log shows this
2022-08-02 08:41:28 [CELL/0] ERR Cell 95d491b4-427b-4f04-9416-85d03bf4b713 failed to create container for instance ea2febea-c37e-4f41-413b-e9df: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/var/vcap/data/rep/shared/garden/trusted_certs" to rootfs at "/etc/cf-system-certificates" caused: mkdir /var/vcap/data/grootfs/store/unprivileged/images/ea2febea-c37e-4f41-413b-e9df/rootfs/etc/cf-system-certificates: permission denied: unknown
From the log, it was not possible to find out that it was an issue with the image built using the bootBuildImage task. After a long investigation, I was able to find out that it was due to the paketo buildpacks that was being used during the image build. Luckily, this configuration can be customized on the gradle task so I had to add the following properties
builder = "paketobuildpacks/builder@sha256:691095c1285bf51ff722bcdf21c22fbdd7fe8421abbe5c68fb23a93fab0651f6"
runImage = "paketobuildpacks/run@sha256:1e2a282085f1b7a0fa1d96bd8509378270ab752c6d8042472a95df2bf5e1d6ea"
This solved the issue. Even though this issue is from a dependency that the bootBuildImage task uses (paketo buildpacks), I had to open it here to be able to track it and so that you folks can forward it to the right repo.
To reproduce the issue - Create a simple Spring boot app and use the bootBuildImage to create an image - Deploy the image on cloud foundry
Comment From: scottfrederick
Thanks for letting us know. We are researching this with the Paketo buildpacks team, and we'll add a link to a Paketo issue to follow when we've identified a root cause.
Comment From: dmikusa
Looking into this a bit, it seems likely due to a change in the runtime base image. I did a bit of testing & the builder image/buildpack images don't seem to trigger the issue, just the runtime image. For example, if I build with just this option set I get a valid image that runs on CF. Is that consistent with what you're seeing @batigoal82?
runImage = "paketobuildpacks/run@sha256:1e2a282085f1b7a0fa1d96bd8509378270ab752c6d8042472a95df2bf5e1d6ea"
My next step is to look at recent changes to the run image & see what might be triggering this. I'll post another update and link to any Paketo issues that this results in.
Comment From: dmikusa
Ok, I tracked this down to a known issue. See here -> https://github.com/paketo-buildpacks/jam/pull/96. There is a fixed merge into the jam tool which is used by our pipelines to publish the run images. The team is going to soon cut a new release of jam and they can then upgrade the pipelines and start using this fix. tl;dr this should very likely be fixed in the next release of the run images.
Comment From: batigoal82
@dmikusa-pivotal Yes, the issue is with the runtime image. I was not sure which runtime image fits with which builder image. So, I set it to the one that I know fits based on previous working images. Thanks for tracking down the issue.