Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sync on-demand: docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json) #2836

Open
tamilhce opened this issue Dec 17, 2024 Discussed in #2811 · 12 comments
Open

sync on-demand: docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json) #2836

tamilhce opened this issue Dec 17, 2024 Discussed in #2811 · 12 comments
Labels
awaiting-answers rm-external Roadmap item submitted by non-maintainers

Comments

@tamilhce
Copy link
Contributor

Discussed in #2811

Originally posted by tamilhce November 28, 2024
I was running zot with following patch version https://github.com/tamilhce/zot/pull/13/ as a pull through for ECR

I have verified that some multi-arch images are working as expected, while others are not. Upon further investigation, I found that container multi-arch images with the type application/vnd.oci.image.index.v1+json (the OCI manifest list for multi-platform images) are not supported. Only multi-arch images with the Docker manifest format, application/vnd.docker.distribution.manifest.list.v2+json (the Docker manifest list for multi-platform images), are supported.

AWS ECR supports both the OCI and Docker manifest formats, whereas Zot only supports the multi-arch/image index of the Docker manifest format.

working - application/vnd.docker.distribution.manifest.list.v2+json

vanan@a-a7ehvnb8bz4k ~> docker manifest inspect xxxx.dkr.ecr.us-east-1.amazonaws.com/xxxx:v1.31.0-1.31.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 921,
         "digest": "sha256:8afaacbf5f90e424d6e70b5ffdc2447112c58031e830e994142fc500064a200f",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 921,
         "digest": "sha256:308b87952a99f44281fadcd7a490b7e8a7cbbb4d3c672ecb417a9b4064a6a316",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}


vanan@a-a7ehvnb8bz4k ~> docker pull zot-dev.xxx.net/xxxxx:v1.31.0-1.31.0
v1.31.0-1.31.0: Pulling from xxxxx
Digest: sha256:ca04f21bc8fcc278acb9bb4f08e359603d3b18da4384822c11e2771480d52a2b
Status: Image is up to date for zot-dev.xxx.net/xxxx:v1.31.0-1.31.0
zot-dev.xxx.net/xxxx:v1.31.0-1.31.0

Not working - application/vnd.oci.image.index.v1+json

vanan@xxxxx ~> docker manifest inspect xxxxxxx.dkr.ecr.us-east-1.amazonaws.com/xxxxxx:xxxxx
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 4841,
         "digest": "sha256:eea456504e848927b80b42427a2b466e25f7ddc226d18765912936001fda6288",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 4843,
         "digest": "sha256:5a6a7e7e93c854dc0f7df8c4ee669c3ba4a30bee27362bb194ba1defbbcb1409",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}


vanan@a-a7ehvnb8bz4k ~> docker pull zot-dev.xxx.net/xxxxx:xxxxx
Error response from daemon: repository zot-dev.xxx.net/xxxxx not found: name unknown: repository name not known to registry


relevant zot container error log 
{"level":"info","repo":"xxxx/xxxxx","xxxxx":"reference","err":"media type is not supported

is this the expected behavior? If it's a bug, I will work on the feature. Looking forward to your valuable reply.
@andaaron
Copy link
Contributor

@tamilhce

  1. Zot supports OCI image indexes in general, and it has been for about 2 years. By support I mean every call mentioned in the oci distribution spec. We have tests for all those scenarios.

  2. The issue you are reporting is specific to zot sync, so I think that should be captured in the issue title.
    Even for zot sync this is really strange, as we do support OCI indexes, and we do have tests for it, such as https://github.com/project-zot/zot/blob/main/pkg/extensions/sync/sync_test.go#L6935

As I asked in #2811, what call is docker actually making to the zot server? Can you share more of the logs?
Do different client tools (such as skopeo for example) work?

@andaaron andaaron changed the title zot not supporting OCI image index(application/vnd.oci.image.index.v1+json) zot sync on-demand: docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json) Dec 17, 2024
@andaaron andaaron changed the title zot sync on-demand: docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json) sync on-demand: docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json) Dec 17, 2024
@tamilhce
Copy link
Contributor Author

tamilhce commented Dec 18, 2024 •
edited
Loading

we have an ECR registry and have configured Zot as a pull-through cache (on-demand).
I observed that performing a docker pull directly from ECR as well as through zot works as expected for the media type "vnd.docker.distribution.manifest.list.v2+json".

However, when attempting a docker pull through Zot for the following OCI manifest, the pull fails:

 ~> docker manifest inspect ecrrepo/appname:1542762016
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 4841,
         "digest": "sha256:eea456504e848927b80b42427a2b466e25f7ddc226d18765912936001fda6288",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 4843,
         "digest": "sha256:5a6a7e7e93c854dc0f7df8c4ee669c3ba4a30bee27362bb194ba1defbbcb1409",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

While docker pull directly from ECR works, the same operation through Zot fails. I do not believe this is a client-side issue. Below are relevant log messages for reference

  • docker command
]> docker pull zot/appname:1590357999
Error response from daemon: repository zot/ecrrepo/appname not found: name unknown: repository name not known to registry
vanan@a-a7ehvnb8bz4k ~ [1]> docker pull ecrrepo/appname:1590357999
1590357999: Pulling from ecrrepo/appname
9088cdb84e39: Already exists
---
c9ca5fb0e249: Pull complete
Digest: sha256:117717899b06c24a7a1008a92e503acbc400626df462d05e8175cb4eb7e38f67
Status: Downloaded newer image for ecrrepo/appname:1590357999
  • zot logs
{"level":"info","module":"http","component":"session","clientIP":"100.80.12.72:22225","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["docker/25.0.6 go/go1.22.5 git-commit/b08a51f kernel/5.15.167-112.165.amzn2.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.5 \\(linux\\))"],"X-Amzn-Trace-Id":["Root=1-6762a921-0c73db7e144f37276bfb5550"],"X-Forwarded-For":["10.12.245.41"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"]},"goroutine":263343,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:13.635698457Z","message":"HTTP API"}
{"level":"info","repository":"appname","reference":"1590357999","goroutine":252054,"caller":"zotregistry.dev/zot/pkg/api/routes.go:2023","time":"2024-12-18T10:51:13.835288399Z","message":"trying to get updated image by syncing on demand"}
{"level":"info","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:562","time":"2024-12-18T10:51:13.835328042Z","message":"getting available client"}
{"level":"info","url":"674283286888.dkr.ecr.us-east-1.amazonaws.com","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/ecr_credential_helper.go:126","time":"2024-12-18T10:51:14.761798199Z","message":"The credentials are close to expiring"}
{"level":"info","url":"674283286888.dkr.ecr.us-east-1.amazonaws.com","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/ecr_credential_helper.go:135","time":"2024-12-18T10:51:14.76182424Z","message":"Refreshing the ECR credentials"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.19.37:40720","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":263403,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:15.945545158Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.19.37:40722","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":263402,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:15.945905526Z","message":"HTTP API"}
{"level":"info","url":"674283286888.dkr.ecr.us-east-1.amazonaws.com","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:170","time":"2024-12-18T10:51:18.190441504Z","message":"Refreshing the upstream remote registry credentials"}
{"level":"info","remote":"ecrrepo","repository":"appname","reference":"1590357999","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:364","time":"2024-12-18T10:51:18.190504252Z","message":"syncing image"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.14.110:44824","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":252128,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:18.943599706Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.14.110:44826","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":252193,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:18.943631903Z","message":"HTTP API"}
{"level":"error","error":"media type is not supported","repository":"appname","reference":"1590357999","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:481","time":"2024-12-18T10:51:21.733418182Z","message":"couldn't get upstream image manifest details"}
{"level":"error","error":"media type is not supported","repository":"appname","reference":"1590357999","goroutine":252054,"caller":"zotregistry.dev/zot/pkg/api/routes.go:2027","time":"2024-12-18T10:51:21.733518301Z","message":"failed to sync image"}
{"level":"info","repo":"appname","1590357999":"reference","err":"media type is not supported","component":"sync","goroutine":252186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/on_demand.go:157","time":"2024-12-18T10:51:21.733556202Z","message":"starting routine to copy image, because of error"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.19.37:25330","method":"HEAD","path":"/v2/appname/manifests/1590357999","statusCode":404,"latency":"8s","bodySize":220,"headers":{"Accept":["application/json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v1+prettyjws"],"User-Agent":["docker/25.0.6 go/go1.22.5 git-commit/b08a51f kernel/5.15.167-112.165.amzn2.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/25.0.5 \\(linux\\))"],"X-Amzn-Trace-Id":["Root=1-6762a921-592230577623e5fb15bfff99"],"X-Forwarded-For":["10.12.245.41"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"]},"goroutine":252054,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:22.148116084Z","message":"HTTP API"}
{"level":"info","repository":"appname","reference":"1590357999","goroutine":263477,"caller":"zotregistry.dev/zot/pkg/api/routes.go:2023","time":"2024-12-18T10:51:22.347902584Z","message":"trying to get updated image by syncing on demand"}
{"level":"info","goroutine":263445,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:562","time":"2024-12-18T10:51:22.347954106Z","message":"getting available client"}
{"level":"info","url":"674283286888.dkr.ecr.us-east-1.amazonaws.com","goroutine":263445,"caller":"zotregistry.dev/zot/pkg/extensions/sync/ecr_credential_helper.go:126","time":"2024-12-18T10:51:23.090439687Z","message":"The credentials are close to expiring"}
{"level":"info","url":"674283286888.dkr.ecr.us-east-1.amazonaws.com","goroutine":263445,"caller":"zotregistry.dev/zot/pkg/extensions/sync/ecr_credential_helper.go:135","time":"2024-12-18T10:51:23.090472688Z","message":"Refreshing the ECR credentials"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.19.37:47038","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":263526,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:25.945641465Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.19.37:47040","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":263527,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:25.945652985Z","message":"HTTP API"}
{"level":"info","url":"674283286888.dkr.ecr.us-east-1.amazonaws.com","goroutine":263445,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:170","time":"2024-12-18T10:51:26.509986629Z","message":"Refreshing the upstream remote registry credentials"}
{"level":"info","remote":"ecrrepo","repository":"appname","reference":"1590357999","goroutine":263445,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:364","time":"2024-12-18T10:51:26.510065991Z","message":"syncing image"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.14.110:58150","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":252196,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:28.943823803Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"100.80.14.110:58148","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30+"]},"goroutine":252195,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2024-12-18T10:51:28.943823799Z","message":"HTTP API"}
{"level":"error","error":"media type is not supported","repository":"appname","reference":"1590357999","goroutine":263445,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:481","time":"2024-12-18T10:51:29.704646557Z","message":"couldn't get upstream image manifest details"}
{"level":"info","repo":"appname","1590357999":"reference","err":"media type is not supported","component":"sync","goroutine":263516,"caller":"zotregistry.dev/zot/pkg/extensions/sync/on_demand.go:157","time":"2024-12-18T10:51:29.704801321Z","message":"starting routine to copy image, because of error"}
{"level":"error","error":"media type is not supported","repository":"appname","reference":"1590357999","goroutine":263477,"caller":"zotregistry.dev/zot/pkg/api/routes.go:2027","time":"2024-12-18T10:51:29.704829701Z","message":"failed to sync image"}

This log line

{"level":"info","module":"http","component":"session","clientIP":"100.80.19.37:25330","method":"HEAD","path":"/v2/appname/manifests/1590357999","statusCode":404,"latency":"8s","bodySize":220,"headers":{"Accept":["application/json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v1+prettyjws"]

indicates that /v2/appname/manifests/1590357999 was not in the accepted media types. However, the image manifest (as confirmed by the manifest inspect I shared earlier) shows that it is in one of the accepted media types. looks like an bug. still I didn't had a chance to review the code.

@andaaron
Copy link
Contributor

{"level":"error","error":"media type is not supported","repository":"appname","reference":"1590357999","goroutine":252056,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:481","time":"2024-12-18T10:51:21.733418182Z","message":"couldn't get upstream image manifest details"} this would be the relevant message. https://github.com/project-zot/zot/blob/main/pkg/extensions/sync/service.go#L426

Question is why it fails to get the image details. It is possible it fails to get the manifest from ecr (instead of having an issue with the manifest content)? Maybe it errors here https://github.com/project-zot/zot/blob/main/pkg/extensions/sync/remote.go#L132 ?

@tamilhce
Copy link
Contributor Author

I have verified the following:

  • This is not a permission issue, as Zot is able to pull other images.
  • The manifest is not malformed because docker pull works on ECR.
  • I’m not sure why retrieving the manifest fails for "mediaType": "application/vnd.oci.image.index.v1+json".

@andaaron
Copy link
Contributor

The strange part is it fails before our checks for manifest media type value.
I need to reproduce myself.

@tamilhce
Copy link
Contributor Author

@andaaron Thanks for linking the relevant code path. I will run a few more tests tomorrow and provide an update here.

I appreciate your quick response—it’s really helpful!

@andaaron
Copy link
Contributor

andaaron commented Dec 18, 2024 •
edited
Loading

Some other limitation about the way sync works with multiarch at the moment:

  1. if periodic sync pulls the image it is usually the image index getting pulled
  2. if on-demand sync pulls the image, the client may explicitly make the request for a specific architecture where the client runs, and only that manifest for that arch would be copied over.
    In both cases 1 and 2, you would end up with the same repo name and same tag after the sync. Even tough the tag points to something else.

There's a PR for replacing the libraries used for sync, but due to limited time on the part of the contributor, it has been dragging for a long time.

@tamilhce
Copy link
Contributor Author

Could you please share the PR link? I’ll give it a try with that branch

@andaaron
Copy link
Contributor

This is the PR, but I don't think it's ready to be used at the moment: #2524

@andaaron
Copy link
Contributor

andaaron commented Dec 26, 2024 •
edited
Loading

Hi @tamilhce,

I have tried to reproduce this issue, with a more basic configuration (as I don't have access to ECR, and also the zot configuration was not provided in the bug description).
Everything went fine. So we need to refine the steps to reproduce.

I tried using 2 zot instances.

1st zot has the config:

{
    "distSpecVersion": "1.1.0",
    "storage": {
        "rootDirectory": "/data/hdd/zot2"
    },
    "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "realm": "zot"
    },
    "log": {
        "level": "debug",
        "output": "/data/hdd/zot2.log"
    },
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "2h"
            }
        }
    }
}

2nd zot has the config:

{
    "distSpecVersion": "1.1.0",
    "storage": {
        "rootDirectory": "/data/hdd/zot3"
    },
    "http": {
        "address": "0.0.0.0",
        "port": "5001",
        "realm": "zot"
    },
    "log": {
        "level": "debug",
        "output": "/data/hdd/zot3.log"
    },
    "extensions": {
        "sync": {
            "enable": true,
            "registries": [
                {
                    "urls": [
                        "http://localhost:5000"
                    ],
                    "onDemand": true,
                    "maxRetries": 6,
                    "tlsVerify": false,
                    "retryDelay": "5m"
                }
            ]
        }
    }
}

I uploaded images to the 1st zot using:
skopeo --insecure-policy copy --dest-tls-verify=false --format=oci --multi-arch=all docker://debian:bookworm docker://127.0.0.1:5000/debian:bookworm

I checked the repo/image/tag is present on zot running on port 5000:

{
    "manifests": [
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "amd64",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "e3f216064528d0ad005524fbafbddfd3115be946",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:cd73f5c112f19fac6d67b49d8982104fcf9c14b4ad69c2658fab8702f61b4430",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 1021
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "amd64",
                "vnd.docker.reference.digest": "sha256:cd73f5c112f19fac6d67b49d8982104fcf9c14b4ad69c2658fab8702f61b4430",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:834412f119e26e44d9de46da5bf43fb95e5f3570bd568d3d429ce384f8f9d653",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "arm32v5",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "96ae5cf33b922b1ebe0b394a0488bd8f41591f2c",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:ee7a87661b1d6ca62839cd582b04738ea6020137b2ce56170f4f74de940caff6",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v5"
            },
            "size": 1041
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "arm32v5",
                "vnd.docker.reference.digest": "sha256:ee7a87661b1d6ca62839cd582b04738ea6020137b2ce56170f4f74de940caff6",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:af80ff0dde8185b4091a20c02feabdf4d2eeef261a466be65779ecf79bd9f0a2",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "arm32v7",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "997dca5116e3abb92ac42a8ce895c96f53a321a4",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:efc40768fcd228826b95493f05f718304349762b8ea5344ddb45032334f421b7",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            },
            "size": 1041
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "arm32v7",
                "vnd.docker.reference.digest": "sha256:efc40768fcd228826b95493f05f718304349762b8ea5344ddb45032334f421b7",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:785674b6ee4ea1eab93cb47872e8ceaebf1cc6f379cdb322020def71583dfc54",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "arm64v8",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "a9c3518f26dd0c08a2625127d5baebeae3eba8e8",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:5d99d20795032654a4d76a464cbd9733a5f0fc3911b6f2a36c1d51211b104afe",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            },
            "size": 1041
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "arm64v8",
                "vnd.docker.reference.digest": "sha256:5d99d20795032654a4d76a464cbd9733a5f0fc3911b6f2a36c1d51211b104afe",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:e6dafd27815785aa7d0dc7651b1d19b45155043d42bcc4ff68ddb9fd9ebc8375",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "i386",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "d378bbefca06de15e86ce242c1882991cb205bc4",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:8ec223efa3a7312781a078ed3557fd5a780d06c6ef047a34f72391f4462c514e",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "386",
                "os": "linux"
            },
            "size": 1017
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "i386",
                "vnd.docker.reference.digest": "sha256:8ec223efa3a7312781a078ed3557fd5a780d06c6ef047a34f72391f4462c514e",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:37f5c2ec14ef40dd026b37e3d059560b2d10c9ab5f001e40bd6130e3d08b139d",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "mips64le",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "f143b23838b95457b64fa753ce2dec022ac0e4d0",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:b5d9524f3fedef64ec6ea84c2e1b3f2ba93b65bce3111625d8990549dc90fd2b",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "mips64le",
                "os": "linux"
            },
            "size": 1029
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "ppc64le",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "8ae00f32f3662f944b54eef8f8032bc2eca984ac",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:0ba38f2a08bbe9100915562c4ce576613ae064c60ccc349483a80f29215f139c",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            },
            "size": 1029
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "ppc64le",
                "vnd.docker.reference.digest": "sha256:0ba38f2a08bbe9100915562c4ce576613ae064c60ccc349483a80f29215f139c",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:ab38f9b910a87bcec1d2a528b1d1bef1bd7895066ed12496f81c2cd3324475ec",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "s390x",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2024-12-23T00:00:00Z",
                "org.opencontainers.image.revision": "f8db2aed5e329e25f933ebfef0505f9665e2f2d4",
                "org.opencontainers.image.source": "https://github.com/debuerreotype/docker-debian-artifacts.git",
                "org.opencontainers.image.url": "https://hub.docker.com/_/debian",
                "org.opencontainers.image.version": "bookworm"
            },
            "digest": "sha256:355db701f3b391739cb2e6b57b3943032c5e4b89062d42e96cee089d5c3f16f7",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "s390x",
                "os": "linux"
            },
            "size": 1021
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "s390x",
                "vnd.docker.reference.digest": "sha256:355db701f3b391739cb2e6b57b3943032c5e4b89062d42e96cee089d5c3f16f7",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:1d6a87f65d816d20c735234d8bc7393be525d154a74e3fa139fc6770b00ded63",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        }
    ],
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "schemaVersion": 2
}

After that I pulled from the 2nd zot.
docker pull 127.0.0.1:5001/debian:bookworm
Which went through fine, and I also checked the tag on the 2nd zot, and it points to the same content as on the 1st zot.

@andaaron
Copy link
Contributor

andaaron commented Dec 26, 2024 •
edited
Loading

I also tried using a build from your branch tamilhce/ecr-cred-helper for the 2nd zot (note there's a go.mod change you haven't committed, and it shows after go mod tidy). The sync was successful, the issue did not reproduce

@rchincha rchincha added the rm-external Roadmap item submitted by non-maintainers label Dec 31, 2024
@tamilhce
Copy link
Contributor Author

tamilhce commented Jan 7, 2025

Hi @andaaron,
Due to family medical reasons, I have not been working for the past couple of weeks. I might need to extend this for another one or two weeks. Once I return, we can connect, or I will share the reproducer with you to move things forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting-answers rm-external Roadmap item submitted by non-maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tamilhce @andaaron @rchincha