support `cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel` #2685

cgwalters · 2022-02-03T14:23:21Z

This is part of coreos/fedora-coreos-tracker#828 conceptually, which was actually in retrospect framed too broadly. Focus shifted to coreos layering but that's really the "user experience" half. Re engineering how we build and ship FCOS (the first issue) still applies.

In particular, I think we should support a cosa init --ostree mode that takes a container image as input and outputs just a container. We may not even generate a builds/ directory, and no meta.json stuff should be created for this.

flowchart TB
    quayprevious["previous ostree container"] --> ostreebuild
    subgraph ostreebuild [cosa ostree build]
    configgit["config git"]-->container
    rpms --> container["ostree container"]
    container --> quay["quay.io"]
    end
    subgraph imagebuild [cosa image build]
    quay --> qemu["qemu image"]
    quay --> metal
    metal --> iso
    qemu --> ami
    qemu --> vsphere
    qemu --> gcp
    end


    imagebuild --> S3

imagebuild -->

Note how the input to "cosa image build" is just the ostree container, not config git (or rpms).

Further, I want to emphasize that the "build ostree container" and "build disk images" can (and would normally be) separate processes. (Now, how testing is integrated here is not depicted, but basically we'd still probably generate a qemu image to sanity test our container builds, but it would be discarded and regenerated by the image build process only once that image had passed other approvals)

The text was updated successfully, but these errors were encountered:

cgwalters · 2022-02-03T14:26:36Z

A specific thing this would really help unblock is reworking our build/CI flow to be more like:

check for changes in input
build new container image
Do sanity checks on that container image as a container (perhaps systemd-in-container even)
Push that container image to registry

The remainder of stuff here could be parallelized/configurable:

Kick off upgrade tests from the previous stable release
Generate a fresh qemu image and run qemu basic tests
Do ISO/metal tests
Do cloud tests

And we could now much more naturally represent stages of CI with container image tags. For example we might push fcos:testing-devel-candidate or so. And then only tag to fcos:testing-devel once some of those tests have passed.

cheesesashimi · 2022-02-03T15:07:48Z

For example we might push fcos:testing-devel-candidate or so. And then only tag to fcos:testing-devel once some of those tests have passed.

This is my favorite part of this. This would enable consumers of these images to get feedback about the overall image state.

Just to clarify, when you say OCI standard keys, are you referring to https://github.com/opencontainers/image-spec/blob/main/annotations.md?

cgwalters · 2022-02-03T15:37:58Z

Yep! Specifically org.opencontainers.image.source and org.opencontainers.image.revision.

For coreos/coreos-assembler#2685 we want to copy e.g. `rpmostree.input-hash` into the container image. Extend the `ExportOpts` struct to support this, and also expose it via the CLI, e.g. `ostree container encapsulate --copymeta=rpmostree.input-hash ...`. And while I was thinking about this...we should by default copy some core ostree keys, such as `ostree.bootable` and `ostree.linux` since they are key pieces of metadata.

cgwalters · 2022-02-03T16:43:17Z

ostreedev/ostree-rs-ext#234 will help this

For coreos/coreos-assembler#2685 we want to copy e.g. `rpmostree.input-hash` into the container image. Extend the `ExportOpts` struct to support this, and also expose it via the CLI, e.g. `ostree container encapsulate --copymeta=rpmostree.input-hash ...`. And while I was thinking about this...we should by default copy some core ostree keys, such as `ostree.bootable` and `ostree.linux` since they are key pieces of metadata.

dustymabe · 2022-02-03T17:34:46Z

I'm not sure I understand the value here. Maybe we can talk about it at the next video community meeting to make it more clear for people.

cgwalters · 2022-02-03T17:46:18Z

ostreedev/ostree-rs#47

Builds on ostreedev/ostree-rs@6d3a69c Part of coreos/coreos-assembler#2685

dustymabe · 2022-02-04T02:01:04Z

ostreedev/ostree-rs#47

Was that a response to me? If so I still don't understand how that answers the question.

Builds on ostreedev/ostree-rs@6d3a69c Part of coreos/coreos-assembler#2685

cgwalters · 2022-02-04T15:12:50Z

Was that a response to me?

Nope, just keeping track of related PRs.

I'm not sure I understand the value here.

I tried to elaborate on all this in coreos/fedora-coreos-tracker#828

The simplest way to say it is that our center of gravity ships much closer to container image builds, and not custom JSON schema stored in a blob store.

Right now the container image is exported from the blob store - this would flip things around; source of truth is a container image. Disk image builds are secondary/derivatives of that.

Builds on ostreedev/ostree-rs-ext#235 Part of coreos/coreos-assembler#2685 Note making use of this will require bumping ostree-ext here.

cgwalters · 2022-02-04T15:16:42Z

coreos/rpm-ostree#3402

Builds on coreos/rpm-ostree#3402 Relates to coreos/coreos-assembler#2685 Basically, source of truth for the CMD moves from being hardcoded in cosa hackiliy to being part of the ostree commit, which means it survives a full round trip of ostree → container → ostree → container

jlebon · 2022-02-04T17:34:41Z

Hmm, also unsure about this. At the end of the day, we'll probably still always want public images sitting in object stores so it's convenient for users/higher-level tools to download and run without involving a container stack. Which means we'd still have something like the builds dir in S3. So there's a lot of force pulling us towards keeping it as canonical too.

cgwalters · 2022-02-04T17:59:16Z

we'll probably still always want public images sitting in object stores so it's convenient for users/higher-level tools to download and run without involving a container stack.

In our world, "images" is an ambiguous term. You're thinking disk/boot images, right? Yes, I agree. Wrapping those in a container is currently a bit of a weird thing to do.

Which means we'd still have something like the builds dir in S3. So there's a lot of force pulling us towards keeping it as canonical too.

I think the interesting angle here more is having disk images come after (follow, derive from) the container builds. But yes, when we go to generate a cosa build, we convert the container back into an ociarchive and store it in S3 as we do currently.

Builds on coreos/rpm-ostree#3402 Relates to coreos/coreos-assembler#2685 Basically, source of truth for the CMD moves from being hardcoded in cosa hackiliy to being part of the ostree commit, which means it survives a full round trip of ostree → container → ostree → container

dustymabe · 2022-03-08T15:06:37Z

I feel like if we're pushing in this direction we should probably have a larger discussion about it. Would you like to bring it up at this week's meeting?

Builds on coreos/rpm-ostree#3402 Relates to coreos/coreos-assembler#2685 Basically, source of truth for the CMD moves from being hardcoded in cosa hackiliy to being part of the ostree commit, which means it survives a full round trip of ostree → container → ostree → container

For now, we need to support having the new format oscontainer in `meta.json`. Part of coreos#2685 And see coreos#2685 (comment) in particular.

For now, we need to support having the new format oscontainer in `meta.json`. Part of #2685 And see #2685 (comment) in particular.

Quite a while ago we added this special case code to kola which learned how to do in-place updates to the weird bespoke "ostree repo in container" OCP/RHCOS-specific container image. A huge benefit of the change to use ostree-native containers is that this approach can now be shared across FCOS/RHCOS. (Also, rpm-ostree natively understands this, so it's much much more efficient and less awkward than the wrappers we had in `pivot` around rpm-ostree) But the benefits get even larger: this issue: coreos#2685 proposes rethinking of our pipeline to more cleanly split up "build OS update container" from "build disk images". With this, it becomes extra convenient to do a flow of: - build OS update container, push to registry - `kola run -p stable --oscontainer quay.io/fcos-devel/testos@sha256...` IOW we're not generating a disk image to test the OS update - we're using the *stable* disk image and doing an in-place update before we run tests. Now...as of right now nothing in the pipeline passes this flag, so the code won't be used (except for manual testing). Suddenly with this, the number of tests we can run roughly *doubles*. For example, we can run e.g. `kola run rpm-ostree` both with and without `--oscontainer`. In most cases, things should be the same. But, I think it will be interesting to try to explictly use this for at least some tests; it's almost a full generalization of the `kola run-upgrades` bits.

Quite a while ago we added this special case code to kola which learned how to do in-place updates to the weird bespoke "ostree repo in container" OCP/RHCOS-specific container image. A huge benefit of the change to use ostree-native containers is that this approach can now be shared across FCOS/RHCOS. (Also, rpm-ostree natively understands this, so it's much much more efficient and less awkward than the wrappers we had in `pivot` around rpm-ostree) But the benefits get even larger: this issue: #2685 proposes rethinking of our pipeline to more cleanly split up "build OS update container" from "build disk images". With this, it becomes extra convenient to do a flow of: - build OS update container, push to registry - `kola run -p stable --oscontainer quay.io/fcos-devel/testos@sha256...` IOW we're not generating a disk image to test the OS update - we're using the *stable* disk image and doing an in-place update before we run tests. Now...as of right now nothing in the pipeline passes this flag, so the code won't be used (except for manual testing). Suddenly with this, the number of tests we can run roughly *doubles*. For example, we can run e.g. `kola run rpm-ostree` both with and without `--oscontainer`. In most cases, things should be the same. But, I think it will be interesting to try to explictly use this for at least some tests; it's almost a full generalization of the `kola run-upgrades` bits.

Part of coreos#2685 I'm looking at replacing the guts of `cosa build ostree` with the new container-native `rpm-ostree compose image`. In order for that to work, we need two things: - The committed overlays from `overlays/` - xref coreos/rpm-ostree#4005 - The rendered `image.json` which is also an overlay now Basically in combination with the above PR, this works now when invoked manually: ``` $ cosa build --prepare-only $ sudo rpm-ostree compose image --cachedir=cache/buildimage --layer-repo tmp/repo src/config/manifest.yaml oci:tmp/fcos.oci ```

Part of #2685 I'm looking at replacing the guts of `cosa build ostree` with the new container-native `rpm-ostree compose image`. In order for that to work, we need two things: - The committed overlays from `overlays/` - xref coreos/rpm-ostree#4005 - The rendered `image.json` which is also an overlay now Basically in combination with the above PR, this works now when invoked manually: ``` $ cosa build --prepare-only $ sudo rpm-ostree compose image --cachedir=cache/buildimage --layer-repo tmp/repo src/config/manifest.yaml oci:tmp/fcos.oci ```

This is a big step towards coreos#2685 I know there's a lot going on with the pipeline, and I don't want to conflict with all that work - but at the same time, in my opinion we are just too dependent on complex Jenkins flows and our bespoke "meta.json in S3". The core of CoreOS *is a container image* now. This new command adds an opinionated flow where one can do: ``` $ cosa init $ cosa build-cimage quay.io/cgwalters/ostest ``` And *that's it* - we do proper change detection, reading and writing from the remote container image. We don't do silly things like storing an `.ociarchive` in S3 when we have native registries available. Later, we can build on this and rework our disk images to derive from that container image, as coreos#2685 calls for. Also in the near term future, I think we can rework `cmd-build` such that it reuses this flow, but outputs to an `.ociarchive` instead. However, this code is going to need a bit more work to run in supermin.

cgwalters · 2022-10-20T13:09:54Z

PR in #3128 which starts the ball rolling here

This is a big step towards coreos#2685 I know there's a lot going on with the pipeline, and I don't want to conflict with all that work - but at the same time, in my opinion we are just too dependent on complex Jenkins flows and our bespoke "meta.json in S3". The core of CoreOS *is a container image* now. This new command adds an opinionated flow where one can do: ``` $ cosa init $ cosa build-cimage quay.io/cgwalters/ostest ``` And *that's it* - we do proper change detection, reading and writing from the remote container image. We don't do silly things like storing an `.ociarchive` in S3 when we have native registries available. Later, we can build on this and rework our disk images to derive from that container image, as coreos#2685 calls for. Also in the near term future, I think we can rework `cmd-build` such that it reuses this flow, but outputs to an `.ociarchive` instead. However, this code is going to need a bit more work to run in supermin.

This will cause us to run through the ostree-native container stack when generating the disk images. Today for RHCOS we're using the "custom origin" stuff which lets us inject metadata about the built source, but rpm-ostree doesn't understand it. With this in the future (particularly after coreos/coreos-assembler#2685) `rpm-ostree status` will show the booted container and *understand it*. We'll have the digest of the OCI archive at least...though that may get changed if it gets converted to docker v2s2 when pushing to a registry... Now in the future what we want is to entirely rework our build pipeline like this: coreos/coreos-assembler#2685

Builds on coreos/rpm-ostree#3402 Relates to coreos/coreos-assembler#2685 Basically, source of truth for the CMD moves from being hardcoded in cosa hackiliy to being part of the ostree commit, which means it survives a full round trip of ostree → container → ostree → container

cgwalters mentioned this issue Feb 3, 2022

container/encapsulate: Support copying commit metadata keys ostreedev/ostree-rs-ext#234

Merged

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Feb 4, 2022

container/encapsulate: Honor ostree.container-cmd

eb42ba3

Builds on ostreedev/ostree-rs@6d3a69c Part of coreos/coreos-assembler#2685

cgwalters added a commit to cgwalters/ostree-rs-ext that referenced this issue Feb 4, 2022

container/encapsulate: Honor ostree.container-cmd

04ecf82

Builds on ostreedev/ostree-rs@6d3a69c Part of coreos/coreos-assembler#2685

cgwalters mentioned this issue Feb 4, 2022

container/encapsulate: Honor ostree.container-cmd ostreedev/ostree-rs-ext#235

Merged

cgwalters mentioned this issue Feb 4, 2022

RFC: A new continuous mechanical FCOS stream coreos/fedora-coreos-tracker#910

Open

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue Feb 4, 2022

treefile: Add container-cmd

a919ec9

Builds on ostreedev/ostree-rs-ext#235 Part of coreos/coreos-assembler#2685 Note making use of this will require bumping ostree-ext here.

cgwalters mentioned this issue Feb 4, 2022

treefile: Add container-cmd coreos/rpm-ostree#3402

Merged

cgwalters mentioned this issue Feb 4, 2022

manifest: Add container-cmd coreos/fedora-coreos-config#1484

Merged

jmarrero added the jira for syncing to jira label Mar 1, 2022

cgwalters mentioned this issue Mar 6, 2022

Optionally record additional information about cosa buildupload s3 location + access to that location in meta.json #2739

Open

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jun 7, 2022

schema: Add entry for base-oscontainer

36f409b

For now, we need to support having the new format oscontainer in `meta.json`. Part of coreos#2685 And see coreos#2685 (comment) in particular.

cgwalters mentioned this issue Jun 7, 2022

schema: Add entry for base-oscontainer #2907

Merged

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jun 7, 2022

schema: Add entry for base-oscontainer

1c3f68f

For now, we need to support having the new format oscontainer in `meta.json`. Part of coreos#2685 And see coreos#2685 (comment) in particular.

cgwalters mentioned this issue Jun 7, 2022

Add remote-build to allow cosa builds remotely using podman remote #2887

Merged

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jun 8, 2022

schema: Add entry for base-oscontainer

b43bc2f

For now, we need to support having the new format oscontainer in `meta.json`. Part of coreos#2685 And see coreos#2685 (comment) in particular.

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jun 8, 2022

schema: Add entry for base-oscontainer

071527b

For now, we need to support having the new format oscontainer in `meta.json`. Part of coreos#2685 And see coreos#2685 (comment) in particular.

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jun 13, 2022

schema: Add entry for baseos-container

54b5d43

For now, we need to support having the new format oscontainer in `meta.json`. Part of coreos#2685 And see coreos#2685 (comment) in particular.

jlebon pushed a commit that referenced this issue Jun 13, 2022

schema: Add entry for baseos-container

432636d

For now, we need to support having the new format oscontainer in `meta.json`. Part of #2685 And see #2685 (comment) in particular.

cgwalters mentioned this issue Jun 13, 2022

MCO-280: extensions: Add container to serve repository with extensions openshift/os#763

Merged

cgwalters mentioned this issue Jul 21, 2022

kola: Change --oscontainer to use new ostree native containers #3004

Merged

cgwalters mentioned this issue Aug 2, 2022

include opinionated but configurable pipeline logic #3008

Open

cgwalters mentioned this issue Sep 12, 2022

build: Add a --prepare-only flag #3091

Merged

cgwalters mentioned this issue Oct 7, 2022

WIP: RHCOS pipeline changes coreos/fedora-coreos-pipeline#652

Closed

cgwalters mentioned this issue Oct 20, 2022

Add new build-cimage command #3128

Open

cgwalters mentioned this issue Oct 20, 2022

formatting of "oscontainer" entries in the schema #3122

Closed

cgwalters mentioned this issue Oct 28, 2022

Make push-container-manifest idempotent #3150

Closed

cgwalters mentioned this issue Nov 7, 2022

kolaTestIso: add miniso-install to UEFI scenarios coreos/coreos-ci-lib#134

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support `cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel` #2685

support `cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel` #2685

cgwalters commented Feb 3, 2022 •

edited

Loading

cgwalters commented Feb 3, 2022

cheesesashimi commented Feb 3, 2022

cgwalters commented Feb 3, 2022

cgwalters commented Feb 3, 2022

dustymabe commented Feb 3, 2022

cgwalters commented Feb 3, 2022

dustymabe commented Feb 4, 2022

cgwalters commented Feb 4, 2022

cgwalters commented Feb 4, 2022

jlebon commented Feb 4, 2022

cgwalters commented Feb 4, 2022

dustymabe commented Mar 8, 2022

cgwalters commented Oct 20, 2022

support cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel #2685

support cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel #2685

Comments

cgwalters commented Feb 3, 2022 • edited Loading

cgwalters commented Feb 3, 2022

cheesesashimi commented Feb 3, 2022

cgwalters commented Feb 3, 2022

cgwalters commented Feb 3, 2022

dustymabe commented Feb 3, 2022

cgwalters commented Feb 3, 2022

dustymabe commented Feb 4, 2022

cgwalters commented Feb 4, 2022

cgwalters commented Feb 4, 2022

jlebon commented Feb 4, 2022

cgwalters commented Feb 4, 2022

dustymabe commented Mar 8, 2022

cgwalters commented Oct 20, 2022

support `cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel` #2685

support `cosa init --ostree docker://quay.io/coreos-assembler/fcos:testing-devel` #2685

cgwalters commented Feb 3, 2022 •

edited

Loading