Unclear behaviour for save/load when using SHA (tag or not)

[MikaelElkiaer]

Description

I will often save my Docker images in a gzipped tar file, in order to load them in a different environment.
Then recently I added checksums along with my tags, in order to pin my versions, while having a version manager update to the latest SHA for the given tag.
This combination is causing issues though.

Actually, using the SHA alone causes this not to work in the same way as using the combination.
AFAICT only the tag alone works.

Reproduce

I have a simple script to reproduce this:

#!/usr/bin/env bash

set -Eeuo pipefail

IMAGE="${1:-docker.io/library/alpine:3.21.3@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe}"

# Pull image
docker pull "$IMAGE"
# Ensure that test runs with pulled image
docker run --pull=never --rm "$IMAGE" echo "If you see this it works!"
# Save pulled image
TAR="$(docker save "$IMAGE" | gzip | base64)"
# Remove local image
docker inspect "$IMAGE" --format='{{.Id}}' | xargs -r docker rmi
# Load saved image to local
echo "$TAR" | base64 --decode | docker load
# See that test fails with saved/loaded image
docker run --pull=never --rm "$IMAGE" echo "If you see this it also works!"

Expected behavior

I expected docker pull to include a reference using the provided tag, even though it is actually pulling via the SHA.

docker version

Client: Docker Engine - Community
 Version:           28.0.1
 API version:       1.48
 Go version:        go1.23.6
 Git commit:        068a01e
 Built:             Wed Feb 26 10:41:16 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.1
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.6
  Git commit:       bbd0a17
  Built:            Wed Feb 26 10:41:16 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        v1.2.4-0-g6c52b3f
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    28.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.21.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.33.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 174
 Server Version: 28.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-28-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.13GiB
 Name: <company PC>
 ID: <>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Registry Mirrors:
  <company mirror>
 Live Restore Enabled: false

Additional Info

I tried both with and without the company mirror for Docker Hub to no avail.

[vvoland]

related:

• Missing error message when pull is run with :TAG and @DIGEST #2856
• Docker SHA-based (digest) pulls should respect and preserve image tag(s) moby/moby#31656

This is a tricky one, as it's not clear what the expected behavior should be, let's consider:

docker pull alpine:3.21@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe

If 3.21 tag no longer points to sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe, I see two potentially expected behavior:

1. Pull sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe, but tag as alpine:3.21 even though it's no longer true
2. Fail the pull because the tag no longer matches the digest

[MikaelElkiaer]

1. implies that it can be verified that the tag did indeed refer to the SHA previously, right?
   Is this always possible? - I assume that cleaning images that are no longer tagged probably could make this hard.
   But that is probably already the case when referring by SHAs today.

[vvoland]

No, we can't reliably verify that, we could only make it possible to tag it locally, regardless of the registry repository.

[vvoland]

I think the second one would make sense - in most cases where you pin a reference to exact tag and digest, I don't think you expect it to change?
In a case where user doesn't care about a tag, they could just use the repo@sha256:digest.

[MikaelElkiaer]

By the way, there might actually be two issues here.

The one being what you addressed - consolidating tag and digest.

However, I think there is a more fundamental problem in that docker save|load does not work as expected even when only using the digest.
Running my example above (with added debug logging), but only referring to the image via digest, I see the following:

λ bash test-image.bash "docker.io/library/alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe"
[DBG] Pulling image: docker.io/library/alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe
docker.io/library/alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe: Pulling from library/alpine
Digest: sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe
Status: Downloaded newer image for alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe
docker.io/library/alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe
[DBG] Running test with pulled image
If you see this it works!
[DBG] Saving image as tarball
[DBG] Removing image from local Dockerd
Untagged: alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe
Deleted: sha256:9ed449c437bfd0ca00973dbbc086fa310e8f7747d5ce78596ceeea177fdd61c8
Deleted: sha256:ef51dabe17eb47d1dab0ed9b3918da01ca30a862359411c99da4bd814513d07f
[DBG] Loading image from tarball
ef51dabe17eb: Loading layer [==================================================>]  6.961MB/6.961MB
Loaded image ID: sha256:9ed449c437bfd0ca00973dbbc086fa310e8f7747d5ce78596ceeea177fdd61c8
[DBG] Running test with loaded image
docker: Error response from daemon: No such image: alpine@sha256:2436f2b3b7d2537f4c5b622d7a820f00aaea1b6bd14c898142472947d5f02abe

Run 'docker run --help' for more information

Even though I pull by the digest, it seems that it is lost when saving the image - also referring by the digest.

[MikaelElkiaer]

I think the second one would make sense - in most cases where you pin a reference to exact tag and digest, I don't think you expect it to change? In a case where user doesn't care about a tag, they could just use the repo@sha256:digest.

One case, though admittedly it is not the strongest of arguments, is when using a version manager - e.g. Renovate [link].
Renovate will pin, and update pinned digests, based on the current tag.
Additionally, it can update tag and digest based on semantic versioning.

[MikaelElkiaer]

Now that I think about it, the problem with consolidating tag and digest is not that big a problem.
Behaviour is actually good as it is right now, in that versions are properly locked by the digest, while Renovate can use the tag to allow for updates outside the current tag.
An error when the two diverge would actually be a problem, as it would force an update in cases where the tag is not idempotent.

The actual issue is the behaviour of docker save|load when using digest.