Action deletes manifests of tagged images

[esune]

I have set-up an action to prune old untagged images as follows:

name: Delete old container images

on:
  schedule:
    - cron: "0 2 * * 2" # This job runs every Tuesday
  workflow_dispatch:

jobs:
  clean-ghcr:
    name: Delete old unused container images
    runs-on: ubuntu-22.04
    if: github.repository_owner == 'bcgov'
    steps:
      - name: Delete container images older than a Month
        uses: snok/[email protected]
        with:
          account: bcgov
          image-names: traction-plugins-acapy, traction-tenant-proxy, traction-tenant-ui
          cut-off: 1month
          tag-selection: untagged
          token: ${{ secrets.GITHUB_TOKEN }}

My expectation would be for the action to only delete images that do not have an explicit tag and are older than one month, however it appears that it is also removing other images as well, possibly layers that are required by tagged images. The symptom we're experiencing is that the image tag is available in the GHCR, however trying to pull the tagged image results in a manifest unknown error.

[sondrelg]

Associated manifests for multi-arch builds do not have tags associated with them. This is partially described in https://github.com/snok/container-retention-policy?tab=readme-ov-file#safely-handling-multi-platform-multi-arch-packages. I hope to fully support this case out of the box in the future, but for now it's up to users to specify which images are safe to delete, if using this with multi-arch packages

[esune]

@sondrelg we are not currently building multi-arch, just amd64 (as far as I can remember) - good to know though.

In our case, it appears that layers required by tagged images are being deleted when untagged is the policy selected (see configuration above).

[sondrelg]

How are you building images? If you run docker inspect ghcr.io/{org}/{package name}:{tag} I suspect you might at least see an associated manifest for new builds - is that right?

[JacobSanford]

@sondrelg Thank you for planning to implement this in the future. It would be really helpful for us. I appreciate your package and the obvious time it takes to maintain.

[esune]

How are you building images? If you run docker inspect ghcr.io/{org}/{package name}:{tag} I suspect you might at least see an associated manifest for new builds - is that right?

We have a github action that builds images and pushes to GHCR. I confirm that it is only amd64 architecture.

[esune]

I just had to manually rebuild and push an image for which I was getting the manifest unknown issue:

$ docker pull ghcr.io/bcgov/traction-plugins-acapy:0.5.0
0.5.0: Pulling from bcgov/traction-plugins-acapy

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview ghcr.io/bcgov/traction-plugins-acapy:0.5.0
manifest unknown

This is the output of the push command:

$ docker push ghcr.io/bcgov/traction-plugins-acapy:0.5.0
The push refers to repository [ghcr.io/bcgov/traction-plugins-acapy]
6c3a6c30ab10: Pushed
a415a1e9b727: Pushed
c44bb3085bb4: Pushed
e2d07531cbd5: Pushed
ef9d8b9e32d4: Pushed
edd31a27bbaa: Pushed
93a758defe24: Pushed
b13ae6128e6c: Layer already exists
f2ffdea5e144: Layer already exists
984442eb3b08: Layer already exists
a07804329048: Layer already exists
5f501f5e0646: Layer already exists
4f9788281efc: Layer already exists
0b28da556a7b: Layer already exists
d71c1661cd45: Layer already exists
5f70bf18a086: Layer already exists
92d34798822d: Layer already exists
30cae9910911: Layer already exists
76f38e225d85: Layer already exists
60830b5bc585: Layer already exists
e8b19277192f: Layer already exists
c11d0a8908db: Layer already exists
6f697f52d485: Layer already exists
0.5.0: digest: sha256:079d6fe9cac1c04933e3103adea4076623c33de39ac0332a84f32f1f615ee360 size: 5131

Several layers were missing, as suspected, however the image is/was still listed in the package list.

Inspecting the image yields the following:

$ docker inspect ghcr.io/bcgov/traction-plugins-acapy:0.5.0
[
    {
        "Id": "sha256:33bca98db8e9025d2c6e83221daed020e53d07f99bca7e1d76258e808620c567",
        "RepoTags": [
            "traction:plugins-acapy",
            "ghcr.io/bcgov/traction-plugins-acapy:0.5.0"
        ],
        "RepoDigests": [
            "ghcr.io/bcgov/traction-plugins-acapy@sha256:079d6fe9cac1c04933e3103adea4076623c33de39ac0332a84f32f1f615ee360"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2024-08-22T23:08:30.392280747Z",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "aries",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/home/aries/.venv/bin:/home/aries/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
                "PYTHON_VERSION=3.9.19",
                "PYTHON_PIP_VERSION=23.0.1",
                "PYTHON_SETUPTOOLS_VERSION=58.1.0",
                "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/e03e1607ad60522cf34a92e834138eb89f57667c/public/get-pip.py",
                "PYTHON_GET_PIP_SHA256=ee09098395e42eb1f82ef4acb231a767a6ae85504a9cf9983223df0a7cbd35d7",
                "HOME=/home/aries",
                "APP_ROOT=",
                "LC_ALL=C.UTF-8",
                "PIP_NO_CACHE_DIR=off",
                "PYTHONUNBUFFERED=1",
                "PYTHONIOENCODING=UTF-8",
                "RUST_LOG=warn",
                "SHELL=/bin/bash",
                "SUMMARY=aries-cloudagent image",
                "DESCRIPTION=aries-cloudagent provides a base image for running Hyperledger Aries agents in Docker.     This image layers the python implementation of aries-cloudagent 0.12.2. Based on Debian Buster."
            ],
            "Cmd": [
                "start",
                "--arg-file",
                "default.yml"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/home/aries",
            "Entrypoint": [
                "/bin/bash",
                "-c",
                "aca-py \"$@\"",
                "--"
            ],
            "OnBuild": null,
            "Labels": {
                "acapy.version": "0.12.2",
                "description": "aries-cloudagent provides a base image for running Hyperledger Aries agents in Docker.     This image layers the python implementation of aries-cloudagent 0.12.2. Based on Debian Buster.",
                "io.k8s.description": "aries-cloudagent provides a base image for running Hyperledger Aries agents in Docker.     This image layers the python implementation of aries-cloudagent 0.12.2. Based on Debian Buster.",
                "io.k8s.display-name": "aries-cloudagent 0.12.2",
                "maintainer": "",
                "name": "aries-cloudagent",
                "org.opencontainers.image.created": "2024-08-01T22:24:37.760Z",
                "org.opencontainers.image.description": "Hyperledger Aries Cloud Agent Python (ACA-Py) is a foundation for building decentralized identity applications and services running in non-mobile environments.",
                "org.opencontainers.image.licenses": "Apache-2.0",
                "org.opencontainers.image.revision": "1631eb9c4eb481bb342f4931f59ebd30d8548da8",
                "org.opencontainers.image.source": "https://github.com/hyperledger/aries-cloudagent-python",
                "org.opencontainers.image.title": "aries-cloudagent-python",
                "org.opencontainers.image.url": "https://github.com/hyperledger/aries-cloudagent-python",
                "org.opencontainers.image.version": "py3.9-0.12.2",
                "summary": "aries-cloudagent image"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 723874167,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/ia8dbohrgzb58fd9iir725din/diff:/var/lib/docker/overlay2/ogvqwji7xduc3f48h22u0odq7/diff:/var/lib/docker/overlay2/inbjpvc2oxfgr0opofnvvb25w/diff:/var/lib/docker/overlay2/m3il7jn8r9yr4w8emdt6fyl78/diff:/var/lib/docker/overlay2/m33b4fubn1vto8onnaed1kko2/diff:/var/lib/docker/overlay2/1qp380gwgi7mkieban77oytvv/diff:/var/lib/docker/overlay2/78e7193382cf1dbd580bc4ae0b0b0029dbf7fb65a9f6f9b88a8b10184c7d56f3/diff:/var/lib/docker/overlay2/541b7b4deee8613c5ebc530888b090cff0291053e21ee851d8138fd698e66921/diff:/var/lib/docker/overlay2/3fd6e54faedd651af1816121f1da7ccd5ac8251380482c139f4b0d7513f805f5/diff:/var/lib/docker/overlay2/049bcc6506a27778c4b4f5ec2e9ea8b034205c38224848a5d624e102b8ad824a/diff:/var/lib/docker/overlay2/0e4f5137030f87ceefe8ad843c845797adfe8c8cf919f535cf7c79e4fa8ca66c/diff:/var/lib/docker/overlay2/19edbf052feefb985d6495055f13584e527b2d0e9f3c1dcc56a35d66047e597d/diff:/var/lib/docker/overlay2/9081c9bd7c3f60da1837ad39f06d3ece59e3970b9518315346d9223ac2d01afd/diff:/var/lib/docker/overlay2/aa090167bc0818f39ac5fe83525deb80fdc4313c30f21022a9ba83d9a51bcb69/diff:/var/lib/docker/overlay2/4558bc77ffcb15b6ae1c655efdeff63a79cd36a6bd38f850a05a0f87330babd5/diff:/var/lib/docker/overlay2/810fcc03d428eca218dece82331e86c1ff7a145ced05a0b58fc0abe787f69a25/diff:/var/lib/docker/overlay2/b3e48de08375115c616da3da20e10e388f35d54ce24cc36b14dd217cfed5daf8/diff:/var/lib/docker/overlay2/de8a872425c82c15b5915f875d340816e31de1e7ab213ddef136c4e7ac4aa3bf/diff:/var/lib/docker/overlay2/9e92c2534cc380171ea304faa2105f73b3a16555d17eb0e2166b58306fe3e368/diff:/var/lib/docker/overlay2/2a0dfd53f9f1a6558cf6997742196d5285fbb4156ba162680cbb6efe7414fa8a/diff:/var/lib/docker/overlay2/55246cda42c8570c257f75a19637b6943885b008532ba18c480e38a2e2d296b4/diff:/var/lib/docker/overlay2/cf0028cc1596c100c8b6b3b0d500de631bc3249551dbd32469c34097f94520c0/diff",
                "MergedDir": "/var/lib/docker/overlay2/x7chyez4ulyeegbhm1wvkbow2/merged",
                "UpperDir": "/var/lib/docker/overlay2/x7chyez4ulyeegbhm1wvkbow2/diff",
                "WorkDir": "/var/lib/docker/overlay2/x7chyez4ulyeegbhm1wvkbow2/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:6f697f52d48595d9c5c3104e2ed3d8e617cb437b485e7396ad5f00726125e52b",
                "sha256:c11d0a8908db993724770b1439e38771d36257a469d74b27f9a9a4e4a5865e64",
                "sha256:e8b19277192f39b8ef414047d9998a4906abb5892c6c703a3e37e222616e6826",
                "sha256:60830b5bc585e5c6fa524bca68accfabe572a655c99b85124cda26b773d357ea",
                "sha256:76f38e225d85b4328917a6613558f620a4715e600f849dc0d7d5cc4b0490136c",
                "sha256:30cae9910911bf55e8273dd8b040151ae87e3dd821b3f3097dc1d2be95b328d7",
                "sha256:92d34798822d7d82d3737011cf92fc46b7ca40908d99aac3da60818994bdc51f",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:d71c1661cd45cfdff79659c570d1ce158423d3a0530b9b60595c9756f8047dfc",
                "sha256:0b28da556a7be5718705316b889cc04d8dea656da472f08c412e68a1a9b263d2",
                "sha256:4f9788281efc424ff55b6657ea392ce0176e0462cee25e2c0c71bb30bde7cb53",
                "sha256:5f501f5e064669ad7fdbf3ac2b44062c4b02f80e6f43e6885ce9ef2fe6495f11",
                "sha256:a07804329048771031259a9e6c12b73fe5a2ce4cda9d33955b203039f062542e",
                "sha256:984442eb3b0809b97ca87ca9c26ea9de038f24e50a69051286cd25a59c8249b9",
                "sha256:f2ffdea5e144ffd86cc9df9babf4bbb85400fa1bd5e4fcfb80ce98a6478bf80a",
                "sha256:b13ae6128e6c41432332bbfbef34c68d9eb1b2dc82ea8f53803456bd2b5b199f",
                "sha256:93a758defe245c942b114c3f928e46d8b11b844e6210a9f5507e14be63f2bdbd",
                "sha256:edd31a27bbaaadf79906d408041ba93c3086c4733583fa45572ead7987528e64",
                "sha256:ef9d8b9e32d40b4400fadea0dbdc6d3e04119d06271b255760ce550815415e7a",
                "sha256:e2d07531cbd56a78fcd99a74209f692b7a8023a71444bc7acbfd0ea1ba0f0bcc",
                "sha256:c44bb3085bb4c621fb7dfce522091948aa908a3b9d97513868fd45b441c2ea1e",
                "sha256:a415a1e9b727ddead2b430a685690b52d6186ae1730dfed0544de8b84a1da821",
                "sha256:6c3a6c30ab10ac73d240d6810c22639fd556e0dbd2070a1e606a177ba1ab973d"
            ]
        },
        "Metadata": {
            "LastTagTime": "2024-08-22T23:08:47.305751036Z"
        }
    }
]