buildah should not cache layers that don't change the filesystem

[mmerickel]

buildah version 1.38.0 (image-spec 1.1.0, runtime-spec 1.2.0)

When using --layers --cache-to ... --cache-from ... I'm seeing buildah store cached layers for every single directive in the file. This includes things like ENV, CMD, LABEL, WORKDIR, ENTRYPOINT. These directives have no effect on the actual filesystem and it definitely takes way more time to cache them than it would be to recompute them.

I noticed this behavior as I'm coming from kaniko where it did not cache these types of changes and buildah stores many more cached layers and is executing slower as a result.

Example dockerfile:

FROM public.ecr.aws/docker/library/python:3.9.20-bookworm

CMD ["/bin/bash"]
ENV PYTHONUNBUFFERED=1
ENV UV_PYTHON_PREFERENCE=system
ENV UV_PYTHON_DOWNLOADS=0

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
<snip />

Example with buildah:

$ buildah build --tag $CI_REGISTRY_IMAGE/$BUILD_IMAGE:$BUILD_TAG --tag $CI_REGISTRY_IMAGE/$BUILD_IMAGE:$CI_COMMIT_SHA --label "CI_PIPELINE_ID=$CI_PIPELINE_ID" --label "CI_JOB_ID=$CI_JOB_ID" --label "GIT_COMMIT=$CI_COMMIT_SHA" --layers --cache-from $CI_REGISTRY_IMAGE/${BUILD_IMAGE}/cache --cache-to $CI_REGISTRY_IMAGE/${BUILD_IMAGE}/cache --cache-ttl $CACHE_TTL --file "$CI_PROJECT_DIR/$BUILD_ROOT/Dockerfile" "$CI_PROJECT_DIR/$BUILD_ROOT"
STEP 1/16: FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
Trying to pull public.ecr.aws/docker/library/python:3.9.20-bookworm...
Getting image source signatures
Copying blob sha256:10c990f494860ae03e096bfd669502cb1d624cbfdb952c9ceb08fcd0c97d34f1
Copying blob sha256:c3cc7b6f04730c072f8b292917e0d95bb886096a2b2b1781196170965161cd27
Copying blob sha256:b2b31b28ee3c96e96195c754f8679f690db4b18e475682d716122016ef056f39
Copying blob sha256:af247aac076473044d24960a352a8ec6f154cf0a28f4fbf35fe5d43b52687ba2
Copying blob sha256:724b30a16eecfb700e7a4734c067e6d5b627a996882d7292df4e613c8c171c0f
Copying blob sha256:2112e5e7c3ff699043b282f1ff24d3ef185c080c28846f1d7acc5ccf650bc13d
Copying blob sha256:73f8983e760efca6be83747dc1f3332d1edf01fd20fc3d08354bbb16a0ece674
Copying config sha256:8223e5d99418aab7262163179079355dd61bff1cb8a60db63c0e8178a1e899ab
Writing manifest to image destination
STEP 2/16: CMD ["/bin/bash"]
--> Pushing cache [example.com/redacted-image/cache]:57242b2a3c75c9c472fdc36c9873df65f4ec44c6567deb9e5013b4b6f6b262c9
--> 2970a5a0d0ca
STEP 3/16: ENV PYTHONUNBUFFERED=1
--> Pushing cache [example.com/redacted-image/cache]:9780e213b9380b82460f8d6c135b58a25ead6fd56157ef44026907942934b0bd
--> abef73982dba
STEP 4/16: ENV UV_PYTHON_PREFERENCE=system
--> Pushing cache [example.com/redacted-image/cache]:8e0b892fea08109b493b4110fde12f485eb08e57b50533ef2fbef42cb2826417
--> 61d4154ebbb2
STEP 5/16: ENV UV_PYTHON_DOWNLOADS=0
--> Pushing cache [example.com/redacted-image/cache]:fd613bcbe7e5904bd5ba1da3e1c9418f856dbcf8b7b5df2621d7d238644fe219
--> a6ad9c718109
STEP 6/16: ENV DEBIAN_FRONTEND=noninteractive
--> Pushing cache [example.com/redacted-image/cache]:76fa56d69fc90b3c7ab8ae4d8f4d8b51db9a107f33468ceed6fb12b383367107
--> cab9d8a740a8
STEP 7/16: RUN apt-get update
<snip />

Example using kaniko:

$ /kaniko/executor --cache=true --context "$CI_PROJECT_DIR/$BUILD_ROOT" --dockerfile "$CI_PROJECT_DIR/$BUILD_ROOT/Dockerfile" --destination "$CI_REGISTRY_IMAGE/$BUILD_IMAGE:$BUILD_TAG" --destination "$CI_REGISTRY_IMAGE/$BUILD_IMAGE:$CI_COMMIT_SHA" --label "CI_PIPELINE_ID=$CI_PIPELINE_ID" --label "CI_JOB_ID=$CI_JOB_ID" --label "GIT_COMMIT=$CI_COMMIT_SHA"
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.9.20-bookworm 
INFO[0000] Retrieving image public.ecr.aws/docker/library/python:3.9.20-bookworm from registry public.ecr.aws 
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.9.20-bookworm 
INFO[0000] Returning cached image manifest              
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.9.20-bookworm 
INFO[0000] Returning cached image manifest              
INFO[0000] Retrieving image manifest public.ecr.aws/docker/library/python:3.9.20-bookworm 
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Building stage 'public.ecr.aws/docker/library/python:3.9.20-bookworm' [idx: '0', base-idx: '-1'] 
INFO[0000] Checking for cached layer example.com/redacted-image/cache:c9f5b52caace2c6c12ea6dc19e012183f94718a9be9d350d06323c888ecbf85a... 
INFO[0000] No cached layer found for cmd RUN apt-get update 
INFO[0000] Unpacking rootfs as cmd RUN apt-get update requires it. 
INFO[0034] CMD ["/bin/bash"]                            
INFO[0034] No files changed in this command, skipping snapshotting. 
INFO[0034] ENV PYTHONUNBUFFERED=1                       
INFO[0034] No files changed in this command, skipping snapshotting. 
INFO[0034] ENV UV_PYTHON_PREFERENCE=system              
INFO[0034] No files changed in this command, skipping snapshotting. 
INFO[0034] ENV UV_PYTHON_DOWNLOADS=0                    
INFO[0034] No files changed in this command, skipping snapshotting. 
INFO[0034] ENV DEBIAN_FRONTEND=noninteractive           
INFO[0034] No files changed in this command, skipping snapshotting. 
INFO[0034] RUN apt-get update                           
INFO[0034] Initializing snapshotter ...                 
INFO[0034] Taking snapshot of full filesystem... 
<snip />

Some rough timings from pipelines I'm seeing:

kaniko uncached build: 4.5 mins
buildah uncached build: 6.5 mins
kaniko fully cached build: 3 mins
buildah fully cached build 5.5 mins