manifest, push: implement `--add-compression` to push with compressed variants. #4912

flouthoc · 2023-07-06T09:55:02Z

This is a feature PR for c/image's EnsureCompressionVariantExists feature merged in below PR:

~~#### Proof of concept to demonstrate the new API which are recently added in c/image and some which still needs to be merged.~~

Adds a new flag --add-compression <compression> to manifest push, which when added will push existing manifest on host while replicating the instances with required compression.

Example usage

./buildah manifest create localhost:5000/list:latest
./buildah build -t test .
./buildah manifest add localhost:5000/list:latest test
# Push with replication for `zstd`
./buildah manifest push --tls-verify=false --all --add-compression zstd localhost:5000/list:latest

Expected pushed manifest list

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d814812aea8237398f0a5c36ee6fabc0dbaa4c1b55453b3b9f87e5a4ac2bd4f1",
      "size": 759,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:de950651c5648b48d8b41fda88245c8fa1060f55125f0387c8b0c2e878733e01",
      "size": 759,
      "annotations": {
        "io.github.containers.compression.zstd": "true"
      },
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }
  ]
}

What type of PR is this?

/kind api-change
/kind bug
/kind cleanup
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake
/kind other

What this PR does / why we need it:

How to verify it

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

manifest, push: implement --add-compression to push with compressed

mtrmac

(Just a quick skim to see how the c/image feature is planned to be used.)

Eventually this should include man pages.
So a simple buildah bud && buildah push will not, with this PR, be able to do this. That’s fine, this PR is valuable as is — it just seems like an enhancement worth tracking.

vendor/github.com/containers/common/libimage/manifests/manifests.go

cmd/buildah/manifest.go

flouthoc · 2023-07-14T06:32:13Z

So a simple buildah bud && buildah push will not, with this PR, be able to do this

@mtrmac I think a simple buildah bud and then buildah push --compression-format <> --compression-level <> can still do it right for a single image, we support that on upstream as of now.

mtrmac · 2023-07-14T18:15:48Z

That creates a zstd-only image, not a dual image compatible with gzip consumers but allowing use of zstd.

openshift-merge-robot · 2023-07-24T09:51:24Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rhatdan · 2023-07-24T14:26:48Z

@flouthoc needs rebase

flouthoc · 2023-07-24T15:09:13Z

@rhatdan Will rebase after this: containers/image#1987

flouthoc · 2023-07-27T18:33:31Z

@giuseppe @mtrmac @vrothberg @rhatdan @nalind PTAL

mtrmac

Code LGTM.

cmd/buildah/manifest.go

docs/buildah-manifest-push.1.md

flouthoc · 2023-07-28T05:53:35Z

@vrothberg @giuseppe PTAL

vrothberg

Code and test LGTM

@edsantiago can you take a look at the test?

docs/buildah-manifest-inspect.1.md

vrothberg · 2023-07-28T06:37:24Z

docs/buildah-manifest-inspect.1.md

+
+**--tls-verify** *bool-value*
+
+Require HTTPS and verification of certificates when talking to container registries (defaults to true).  TLS verification cannot be used when talking to an insecure registry.


Nit: the new flags are not tested in the commit adding them but the one after.

I was not planning to add these flags but they were needed for the --add-compression test, I can add independent tests for these but they will be just clone of --add-compression test without the --add-compression flag, I decided not to add because of duplication since these two flags are already getting tested in the test added for --add-compression. But I could be wrong 😅

vrothberg · 2023-07-28T06:38:18Z

docs/buildah-manifest-push.1.md

@@ -18,6 +18,11 @@ The list image's ID and the digest of the image's manifest.

 ## OPTIONS

+**--add-compression** *compression*
+
+If specified while pushing option makes sure that requested compression variant exists for each platform in the manifest list.


This sentence is a bit hard to brain parse. I'd drop while pushing since we're already in the buildah-push man page.

We should also elaborate how in interacts with --compression and that the ones specified here are in addition to the default or --compression one.

Could you check if it looks any better now.

tests/bud.bats

edsantiago

I'm going to assume that "testing for gzip" == "null annotation". I think that should be clearer in the comments.

There's a lot of unnecessary duplication in the test, it could maybe be refactored into a table. And, when possible, it's better to do exact matches instead of substring.

This is just a quick glance, sorry, gotta head out.

openshift-ci · 2023-07-31T12:31:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, flouthoc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [flouthoc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

flouthoc · 2023-07-31T12:44:36Z

@containers/buildah-maintainers PTAL

rhatdan · 2023-07-31T12:57:11Z

cmd/buildah/manifest.go

@@ -696,6 +702,9 @@ func manifestAnnotateCmd(c *cobra.Command, args []string, opts manifestAnnotateO
 }

 func manifestInspectCmd(c *cobra.Command, args []string, opts manifestInspectOpts) error {
+	if err := auth.CheckAuthFile(opts.authfile); err != nil {


Can you wrap this in a check to see if the user modified it?

rhatdan · 2023-07-31T12:57:57Z

LGTM other the check to make sure user set the authfile.

Add flags for `manifest inspect` i.e `--tls-verify` and `--authfile` Signed-off-by: Aditya R <[email protected]>

Adds support for `--add-compression` which accepts multiple compression formats and when used it will add all instances in a manifest list with requested compression formats. Signed-off-by: Aditya R <[email protected]>

rhatdan · 2023-07-31T15:03:45Z

/lgtm
/hold

flouthoc · 2023-07-31T16:45:09Z

/hold cancel

flouthoc · 2023-07-31T17:39:04Z

@lsm5 Some rpm-builds are failing, just informing.

lsm5 · 2023-07-31T17:40:43Z

@flouthoc yup, there was a second issue on c9s. I've filed bz and I have a temp fix for podman. I'll create one for buildah too.

rhatdan · 2023-08-01T21:11:21Z

If I want to push zstd:chunked only, now what do I do? Can I set this up in containers.conf to do this all the time?

flouthoc · 2023-08-02T01:00:35Z

Can I set this up in containers.conf to do this all the time

Not as of now but I think we can create a field in container.conf for default value of --add-compression WDYT ? ( created a PR: containers/common#1593 )

If I want to push zstd:chunked only

I think zstd:chunked is still waiting for this : containers/image#1980 and if only zstd:chunked is needed then you can use simple --compression flag.

Allows users to set default value of `AddCompression` to Engine table so users can use containers/buildah#4912 by default. Closes: containers/buildah#4912 (comment) Signed-off-by: Aditya R <[email protected]>

rhatdan · 2023-08-02T12:52:57Z

No. In Fedora 39, the default for pushing to registries has to be "gzip,zstd:chunked" Then Podman can take advantage of zstd. And old docker will continue to work.
In Fedora 40 we ship to just zstd:chunked. But this setting has to be set in containers.conf so distributions can override it.

mtrmac · 2023-08-02T19:15:48Z

the default for pushing to registries

I mean, we are not at that point at all yet. This is manifest push, with an explicit sequence of commands.

A simple podman push / buildah push (optionally!) creating a multi-instance image definitely makes sense, but just doesn’t exist right now. That path is also different in that the source is always uncompressed, so the difference between --compression and --add-compression is not as pronounced.

rhatdan · 2023-08-03T10:42:09Z

Ok as long as I get my eventual goal, I am fine with this.

Allows users to set default value of `AddCompression` to Engine table so users can use containers/buildah#4912 by default. Closes: containers/buildah#4912 (comment) Signed-off-by: Aditya R <[email protected]>

flouthoc marked this pull request as draft July 6, 2023 09:55

openshift-ci bot added approved do-not-merge/work-in-progress labels Jul 6, 2023

flouthoc force-pushed the replicate-zstd branch 2 times, most recently from 56bda6b to d9e34fa Compare July 13, 2023 08:24

flouthoc mentioned this pull request Jul 13, 2023

copy: implement instanceCopyClone for zstd compression containers/image#1987

Merged

mtrmac reviewed Jul 13, 2023

View reviewed changes

vendor/github.com/containers/common/libimage/manifests/manifests.go Outdated Show resolved Hide resolved

vendor/github.com/containers/common/libimage/manifests/manifests.go Outdated Show resolved Hide resolved

cmd/buildah/manifest.go Outdated Show resolved Hide resolved

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2023

flouthoc force-pushed the replicate-zstd branch from d9e34fa to 6c5ccbc Compare July 27, 2023 18:30

flouthoc changed the title ~~[POC] manifest, push: implement --replicate with compression while pushing~~ manifest, push: implement --add-compression with compression while pushing Jul 27, 2023

flouthoc marked this pull request as ready for review July 27, 2023 18:31

flouthoc requested a review from mtrmac July 27, 2023 18:32

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 27, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 27, 2023

flouthoc force-pushed the replicate-zstd branch from 6c5ccbc to 56e6308 Compare July 27, 2023 18:35

flouthoc changed the title ~~manifest, push: implement --add-compression with compression while pushing~~ manifest, push: implement --add-compression to push with compressed variants. Jul 27, 2023

mtrmac reviewed Jul 27, 2023

View reviewed changes

cmd/buildah/manifest.go Show resolved Hide resolved

docs/buildah-manifest-push.1.md Outdated Show resolved Hide resolved

flouthoc force-pushed the replicate-zstd branch from 56e6308 to 08ada2c Compare July 28, 2023 03:24

vrothberg reviewed Jul 28, 2023

View reviewed changes

flouthoc force-pushed the replicate-zstd branch 2 times, most recently from c7b2e11 to 46ac671 Compare July 28, 2023 07:33

edsantiago reviewed Jul 28, 2023

View reviewed changes

tests/bud.bats Outdated Show resolved Hide resolved

edsantiago reviewed Jul 28, 2023

View reviewed changes

rhatdan reviewed Jul 31, 2023

View reviewed changes

flouthoc added 2 commits July 31, 2023 20:24

manifest/inspect: add support for tls-verify and authfile

d27a823

Add flags for `manifest inspect` i.e `--tls-verify` and `--authfile` Signed-off-by: Aditya R <[email protected]>

manifest/push: add support for --add-compression

b9ac8d5

Adds support for `--add-compression` which accepts multiple compression formats and when used it will add all instances in a manifest list with requested compression formats. Signed-off-by: Aditya R <[email protected]>

flouthoc force-pushed the replicate-zstd branch from edc068b to b9ac8d5 Compare July 31, 2023 14:54

openshift-ci bot assigned rhatdan Jul 31, 2023

openshift-ci bot added do-not-merge/hold lgtm labels Jul 31, 2023

openshift-ci bot removed the do-not-merge/hold label Jul 31, 2023

rhatdan merged commit 8af2dc4 into containers:main Jul 31, 2023
31 of 33 checks passed

This was referenced Jul 31, 2023

libimage/ManifestPushOptions: add AddCompression for Manifest.PushOptions containers/common#1590

Merged

manifest, push: implement --add-compression to push with compressed variants. containers/podman#19468

Merged

flouthoc mentioned this pull request Aug 2, 2023

containers.conf: add field for AddCompression to Engine table containers/common#1593

Merged

edsantiago mentioned this pull request Sep 12, 2023

buildah manifest inspect --tls-verify=false Error: unknown flag: --tls-verify #5038

Closed

github-actions bot added the locked - please file new issue/PR label Nov 2, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

manifest, push: implement `--add-compression` to push with compressed variants. #4912

manifest, push: implement `--add-compression` to push with compressed variants. #4912

flouthoc commented Jul 6, 2023 •

edited

Loading

mtrmac left a comment

flouthoc commented Jul 14, 2023

mtrmac commented Jul 14, 2023

openshift-merge-robot commented Jul 24, 2023

rhatdan commented Jul 24, 2023

flouthoc commented Jul 24, 2023

flouthoc commented Jul 27, 2023 •

edited

Loading

mtrmac left a comment

flouthoc commented Jul 28, 2023

vrothberg left a comment

vrothberg Jul 28, 2023

flouthoc Jul 28, 2023

vrothberg Jul 28, 2023

flouthoc Jul 28, 2023

edsantiago left a comment

openshift-ci bot commented Jul 31, 2023

flouthoc commented Jul 31, 2023

rhatdan Jul 31, 2023

rhatdan commented Jul 31, 2023

rhatdan commented Jul 31, 2023

flouthoc commented Jul 31, 2023

flouthoc commented Jul 31, 2023

lsm5 commented Jul 31, 2023

rhatdan commented Aug 1, 2023

flouthoc commented Aug 2, 2023 •

edited

Loading

rhatdan commented Aug 2, 2023

mtrmac commented Aug 2, 2023

rhatdan commented Aug 3, 2023


		--tls-verify bool-value

		Require HTTPS and verification of certificates when talking to container registries (defaults to true). TLS verification cannot be used when talking to an insecure registry.

manifest, push: implement --add-compression to push with compressed variants. #4912

manifest, push: implement --add-compression to push with compressed variants. #4912

Conversation

flouthoc commented Jul 6, 2023 • edited Loading

What type of PR is this?

What this PR does / why we need it:

How to verify it

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

mtrmac left a comment

Choose a reason for hiding this comment

flouthoc commented Jul 14, 2023

mtrmac commented Jul 14, 2023

openshift-merge-robot commented Jul 24, 2023

rhatdan commented Jul 24, 2023

flouthoc commented Jul 24, 2023

flouthoc commented Jul 27, 2023 • edited Loading

mtrmac left a comment

Choose a reason for hiding this comment

flouthoc commented Jul 28, 2023

vrothberg left a comment

Choose a reason for hiding this comment

vrothberg Jul 28, 2023

Choose a reason for hiding this comment

flouthoc Jul 28, 2023

Choose a reason for hiding this comment

vrothberg Jul 28, 2023

Choose a reason for hiding this comment

flouthoc Jul 28, 2023

Choose a reason for hiding this comment

edsantiago left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 31, 2023

flouthoc commented Jul 31, 2023

rhatdan Jul 31, 2023

Choose a reason for hiding this comment

rhatdan commented Jul 31, 2023

rhatdan commented Jul 31, 2023

flouthoc commented Jul 31, 2023

flouthoc commented Jul 31, 2023

lsm5 commented Jul 31, 2023

rhatdan commented Aug 1, 2023

flouthoc commented Aug 2, 2023 • edited Loading

rhatdan commented Aug 2, 2023

mtrmac commented Aug 2, 2023

rhatdan commented Aug 3, 2023

manifest, push: implement `--add-compression` to push with compressed variants. #4912

manifest, push: implement `--add-compression` to push with compressed variants. #4912

flouthoc commented Jul 6, 2023 •

edited

Loading

flouthoc commented Jul 27, 2023 •

edited

Loading

flouthoc commented Aug 2, 2023 •

edited

Loading