copy: replicate manifest list instances with zstd compression on ReplicateZstd:true

[flouthoc]

Extends copy to support a new operation when copying a manifest list from src to dest i.e ReplicateZstd. When ReplicateZstd is set to true and copy operation is performed for a manifest list then a copy for all the image will be created and added to the manifest list with compression as zstd and appropriate annotation i.e "io.github.containers.compression.zstd": "true" will be set on each image.

The CLI tools must use ReplicateZstd flag in order to use this new feature.

Example with skopeo

• Here src docker://localhost:5000/list only contains two gzip compressed image but while copying skopeo will replicated zstd instances as well

$ ./skopeo copy --all --src-tls-verify=false --dest-tls-verify=false docker://localhost:5000/list docker://localhost/list
Getting image list signatures
Copying 4 of 4 images in list
Copying image sha256:915468dd2aae5e243dbb8d38afc445e8971f7076deade33b81128fdfa42a17a4 (1/4)
Getting image source signatures
Copying blob f5763aacc581 done
Copying blob 0aa932d79126 done
Copying config ba4505248d done
Writing manifest to image destination
Storing signatures
Copying image sha256:f08c3c433a2c73ba643800326a825493a862a438857a227f01ae63e0aa6f3896 (2/4)
Getting image source signatures
Copying blob 1d86523f4719 done
Copying blob 0aa932d79126 done
Copying config 1ea47b71b3 done
Writing manifest to image destination
Storing signatures
Copying image sha256:915468dd2aae5e243dbb8d38afc445e8971f7076deade33b81128fdfa42a17a4 (3/4)
Getting image source signatures
Copying blob f5763aacc581 done
Copying blob 0aa932d79126 done
Copying config ba4505248d done
Writing manifest to image destination
Storing signatures
Copying image sha256:f08c3c433a2c73ba643800326a825493a862a438857a227f01ae63e0aa6f3896 (5/4)
Getting image source signatures
Copying blob 1d86523f4719 done
Copying blob 0aa932d79126 done
Copying config 1ea47b71b3 done
Writing manifest to image destination
Storing signatures
Writing manifest list to image destination
Storing list signatures

• output

$ skopeo inspect --tls-verify=false --raw docker://localhost/list | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:915468dd2aae5e243dbb8d38afc445e8971f7076deade33b81128fdfa42a17a4",
      "size": 759,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f08c3c433a2c73ba643800326a825493a862a438857a227f01ae63e0aa6f3896",
      "size": 759,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:2a8521136269bf9113cdf9088288a07cce52850d15b958b43aad6da5dc0115fb",
      "size": 759,
      "annotations": {
        "io.github.containers.compression.zstd": "true"
      },
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:4feb1d9bf33b2f771dbee9b5ece774f1b6b4dc49b82bce686e47afd13686dcb5",
      "size": 759,
      "annotations": {
        "io.github.containers.compression.zstd": "true"
      },
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }
  ]
}

[flouthoc]

@mtrmac @giuseppe PTAL my first iteration approach on this one. ( also just wondering if skopeo failure are because of the PR or unrelated )

[TomSweeneyRedHat]

nit of a nit

Suggested change

	// and appropriate annotations configured for each newly compressed image the manifest list.
	// and appropriate annotations configured for each newly compressed image in the manifest list.

[mtrmac]

Thanks!

I think this is a good top-level view of what will be necessary, but let’s build the infrastructure in smaller steps.

• (Split copy/copy.go into three files #1878 A separate PR to move copyOneImage and immediate callees, completely unchanged, into copy/single.go, and copyMultipleImages etc. to copy/multiple.go? It might be time to split the copy.go code further.)
• copy: move c.compression* to imageCopier #1881 A separate PR to move the copier.compression* data into imageCopier, and consuming it from there.
• (copy/multiple: split selection of images to be copied in copyMultipleImages #1982 Maybe a separate PR to split the “what to copy and how” logic of copyMultipleImages into a separate function, with unit tests for the existing CopySpecificImages logic.)
• copy/multiple: use more flexible EditInstances instead of UpdateInstances #1883 A separate PR to have copy use a a new internal-only internal/manifest.List.EditInstances
• manifest: prepare internal EditInstances #1896 A PR (possibly combined with the previous one) to add the necessary functionality to EditInstances; also designed to make Add copy option to strip sparse manifests #1707 possible
• A separate PR to ensure existing callers of multi-image copies with DestinationCtx.CompressionFormat asking for zstd should (only if the result actually has a zstd layer??) result in that image having the zstd annotation added.
• … and then we should have a smaller-scale task ahead of us, focusing on what to copy/convert, and on the semantics/optimizations of individual layer conversion.

[mtrmac]

( also just wondering if skopeo failure are because of the PR or unrelated )

It’s almost certainly this PR. (I’m afraid the output is very noisy, but search for FAIL:):

    --- FAIL: TestCopy/TestCopyNoneWithManifestList (0.30s)
        utils.go:68: Running skopeo copy --multi-arch=index-only docker://registry.fedoraproject.org/fedora-minimal dir:/tmp/TestCopyTestCopyNoneWithManifestList3756920598/001
        utils.go:70: 
            	Error Trace:	/var/tmp/go/src/github.com/containers/skopeo/integration/utils.go:70
            	            				/var/tmp/go/src/github.com/containers/skopeo/integration/copy_test.go:148
            	Error:      	Received unexpected error:
            	            	exit status 2
            	Test:       	TestCopy/TestCopyNoneWithManifestList
            	Messages:   	Getting image list signatures
            	            	Copying 0 of 0 images in list
            	            	panic: runtime error: index out of range [0] with length 0
            	            	
            	            	goroutine 1 [running]:
            	            	github.com/containers/image/v5/copy.(*copier).copyMultipleImages(0xc0001e6480, {0x55649775b8b8, 0xc000126000}, 0xc0006665a0?, 0xc0007116c8, 0x1924924?)
            	            		/var/tmp/go/src/github.com/containers/skopeo/vendor/github.com/containers/image/v5/copy/copy.go:528 +0x231f
            	            	github.com/containers/image/v5/copy.Image({0x55649775b8b8, 0xc000126000}, 0x0?, {0x5564977611d8, 0xc0002e4c20}, {0x556497761238?, 0xc0003b8060?}, 0xc000149078?)

[mtrmac]

This will be fully implemented when #1987 is merged, closing this prototype.

Thanks for doing all that work!