Metadata descriptors

[LaurentGoderre]

This PR allows outputting all the descriptors in the metadata file. This is how it would look:

"containerimage.descriptors": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:09ebf864af696306655197662bac5fc5d249e273223bf0c02df053c5b969e723",
      "size": 1607,
      "annotations": {
        "org.opencontainers.image.created": "2024-10-28T13:41:56Z"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:465f515dfba9a241c3af16b4154e4161db1e8de1055b753434874158e7ae05d9",
      "size": 480,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:e75ff1d806b99c41ff0f797f6800a85546d5ebf447d69796ec1db166809a41a2",
      "size": 585
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6769e746bda352f77bbc473606b8e55bed52a2e091d7bba0d953aba8915410ac",
      "size": 480,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:1c1104847de44646826edc258cfc23c4955f05ab2110d6aa1944ee283f993bb2",
      "size": 585
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:a8287ced569f2ab73e293791fcab87478106a5b87eda2567ada9341c675f52c2",
      "size": 838,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:465f515dfba9a241c3af16b4154e4161db1e8de1055b753434874158e7ae05d9",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:c844cf966bdfe7d63e4b87f69b66238fe0e8e42539faa3240258b07f3e85d45d",
      "size": 838,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:6769e746bda352f77bbc473606b8e55bed52a2e091d7bba0d953aba8915410ac",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
]

[tonistiigi]

Check the CI errors

[tonistiigi]

Don't quite understand what is happening here. Why are we marshalling just to unmarshal the same value again.

After parsing, check that these are real descriptors, with real digest/size/mediatype and that the length matches.

[LaurentGoderre]

TBH, I just followed the previous block:

https://github.com/moby/buildkit/blob/master/cmd/buildctl/build_test.go#L156-L158

[tonistiigi]

Ok, this looks to be (convoluted) way to convert interface{} backed by map to a typed struct. Usually, a better approach would be to parse into map[string]json.RawMessage.

If you add a comment for this, then you can leave the conversion as is but do add the checks to make sure that the descriptors are real and contain correct values. Ideally, I think this should also be checked for both single-arch and multi-arch results as looks that the expected output is different.

[tonistiigi]

Shouldn't this be just mfstDesc. Mixing manifest and config descriptors together does not make a predictable result. And how would one even connect the ones that are for the same image.

[tonistiigi]

Testing

For single arch build I get:

  "containerimage.descriptor": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:4399329e7d074f75334ae57e17600c6b694752ddc995925bd2e8dead2e527a23",
    "size": 672,
    "annotations": {
      "org.opencontainers.image.created": "2024-11-22T22:47:59Z"
    },
    "platform": {
      "architecture": "arm64",
      "os": "linux"
    }
  },
  "containerimage.descriptors": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:4399329e7d074f75334ae57e17600c6b694752ddc995925bd2e8dead2e527a23",
      "size": 672,
      "annotations": {
        "org.opencontainers.image.created": "2024-11-22T22:47:59Z"
      },
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:78b9c55f4bcdac3f3768bb6d9ce4947c3098140595f7e8f13b3dba927206e1bf",
      "size": 833
    }
  ],
  "containerimage.digest": "sha256:4399329e7d074f75334ae57e17600c6b694752ddc995925bd2e8dead2e527a23"

So there isn't much difference between old field and new one. Except for the config descriptor.

Multi-arch:

 "containerimage.descriptor": {
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "digest": "sha256:d633778c92d58e0210fea8f959f6e6d8e4dad8d99431abed5ee7fd5ffa23491d",
    "size": 1607,
    "annotations": {
      "org.opencontainers.image.created": "2024-11-22T22:49:09Z"
    }
  },
  "containerimage.descriptors": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:d633778c92d58e0210fea8f959f6e6d8e4dad8d99431abed5ee7fd5ffa23491d",
      "size": 1607,
      "annotations": {
        "org.opencontainers.image.created": "2024-11-22T22:49:09Z"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d2b38a8c974cd88dae45c6f9ea13cf1b05c337c1e69a0547af8a3c463489c797",
      "size": 672,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:6fe24fa44bfb9af9e146e775f6a72503dd3f9bfe7678d9f86defa4b84c8f0d39",
      "size": 817
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:9cc1e9b2e46383939de3f05886baf69c0132136dfeb0c767a4e069659dac1fe7",
      "size": 672,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:292cad1b8f1528fea3626c83415e892fb7e399aa91aad02e9c9bd05951f4cf80",
      "size": 818
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:fd159f9b8bec20a69d5fcc5002c4de98103fb98ae6c5d4a50e13dcc77ebe4f4c",
      "size": 565,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:d2b38a8c974cd88dae45c6f9ea13cf1b05c337c1e69a0547af8a3c463489c797",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:60f696f5a9027bf7596d622a3f7c6a145b97d03e3e7ef4a40f8508b250f49ca4",
      "size": 565,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:9cc1e9b2e46383939de3f05886baf69c0132136dfeb0c767a4e069659dac1fe7",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ],
  "containerimage.digest": "sha256:d633778c92d58e0210fea8f959f6e6d8e4dad8d99431abed5ee7fd5ffa23491d"

This one has root manifest (duplicate), single-arch manifests, configs for single-arch, attestations manifest, (but no configs for attestations). I guess my initial thought would have been that the old field contains the descriptor of the root, and new field contains the descriptors immediately under the root (without the root itself and without the configs). I don't know if that helps with the use-case you are trying to use it though, but atm. there doesn't seem to be a very clear definition of which descriptors are appended to the result and which are skipped. And as mentioned in the previous comment, it isn't very clear how these additional config descriptors should be used as it cannot be determined what manifest or platform they belong to.

[LaurentGoderre]

Its usefulness is more obvious when using attestation manifest and tags. Here is the output I get for a single arch

"containerimage.descriptor": {
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "digest": "sha256:9503904dd20d71f165afb82819327f30ce1c9d0d0d8216589877c302f139c4cf",
    "size": 856,
    "annotations": {
      "org.opencontainers.image.created": "2024-11-25T14:48:37Z"
    }
  },
  "containerimage.descriptors": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:9503904dd20d71f165afb82819327f30ce1c9d0d0d8216589877c302f139c4cf",
      "size": 856,
      "annotations": {
        "org.opencontainers.image.created": "2024-11-25T14:48:37Z"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ec637423e72cdd0b21aa8c80639d5810fb9c7b2c61a73c848f63353b4c8043f1",
      "size": 1052,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:10166aa54184f653461204fda1cb83acd8cbb21d2fa0f35901f8852e83c8af4a",
      "size": 6215
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:2cf7dc5a44a200a07c57ddbe0f44b52ba26f8e96e57b06d735d8d6d030a01eb8",
      "size": 840,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:ec637423e72cdd0b21aa8c80639d5810fb9c7b2c61a73c848f63353b4c8043f1",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ],