Packaging OKD binaries for Fedora

[omenos]

Continuation of the WG 10/26 call

Proposal

This discussion is about packaging the OKD userspace applications, namely openshift-install and oc, for distribution within Fedora.

Current State

Presently I have a trial run of this available in a Fedora COPR. The packages themselves do not follow Fedora packaging guidelines in relation to Golang projects, and bypass the standard build process for the applications (e.g. using the hack/ scripts and Makefiles) in favor of direct go build calls. Also worth noting the binaries have been renamed from their upstream openshift-install and oc names to okd-install and okd in order to prevent name clashing with the OpenShift binaries. This carries over into the provided bash completion scripts.

What has been built so far is the 4.8.0.okd-2021-10-24 release for x86_64 and aarch64. I'm working on getting aarch64 and potentially other builds going, but emulation is slow going for local work. I was trying to avoid spamming COPR.

These binaries are also pretty untested outside of seeing if they run in general. Would be great to have feedback on their actual functionality. Unsure of any quirks arising from how they are built.

Resource	Location
COPR	mroche/okd
Git	Pagure
Installer Source	Vadim's fork
Client Source	Upstream oc

Steps for usage

Enable the COPR (use any following method):

dnf copr enable mroche/okd

Install the packages:

dnf install okd-installer okd-client

Might rename okd-installer to okd-install...

Issues

Where's kubectl?
This is already available in the kubernetes-client and origin-clients packages. Is it worth conflicting on this? kubectl isn't built with the current package process.

We already have Origin packages in Fedora which provide oc.
Origin and its applications are for the 3.x branch of OpenShift, OKD handles 4.x. It also doesn't include openshift-install. Maybe worth submitting for the removal of Origin packages? Currently shipping in F34:

origin
origin-clients
origin-cluster-capacity
origin-docker-excluder
origin-dockerregistry
origin-excluder
origin-hyperkube
origin-hypershift
origin-master
origin-node
origin-pod
origin-sdn-ovs
origin-service-catalog
origin-template-service-broker
origin-tests
origin-web-console

How are updates to the packages going affect users?
As noted in the call, there is a coupling going on between openshift-install and the OKD+FCOS release streams it uses. There is the ability to use runtime override environment variables OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE and OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE to tweak version selection, but the installer itself is not completely release agnostic in what may be considered a layman perception. @LorbusChris and @travier provided more information on this on-call than I could accurately paraphrase.

Fundamentally, will using (going to adjust versioning here for simplicity) a 4.8.3 installer cause problems when creating/modifying/destroying clusters < 4.8.3 and clusters > 4.8.3? Not considering cross-minor versions in this scenario.

OKD Client lacks releaseClientVersion in the version output!
Due to the fact I'm not using the Makefile structure for oc, the post-build binary modification to inject this key-value isn't occurring (from what I can tell). I haven't gotten the chance to dig too deep into the build process to figure out how to mimic this myself, but okd version does return the correct information from the other version info provided.

Ideas

• Packaging latest available in time for feature freeze for a given Fedora release.
• Use Modularity to provide streams for each release version (e.g. 4.8, 4.9, etc) with enforced stream EOL.
• Copy some information/steps from the oc.spec

[vrutkovs]

What is the benefit of this packaging? Do we have Fedora package request somewhere?

[vrutkovs]

Fundamentally, will using (going to adjust versioning here for simplicity) a 4.8.3 installer cause problems when creating/modifying/destroying clusters < 4.8.3 and clusters > 4.8.3?

Within micro version (4.8.x) you're most likely safe (you may hit bugs on some corner-case install and/or deprovision).

You'll certainly need a new installer to install a different minor version (4.x)

[vrutkovs]

How automated is the new package release procedure?

We already have quite a long list of manual actions to do for the release and I'd prefer to avoid additional ones which may not be necessary.

[travier]

I have the same comments and concerns as @vrutkovs here. Those are more specifically for openshift-install/okd-install although they also apply to packaging oc/okd but in a lesser way:

• What are the benefits of packaging for Fedora (or any other distribution)?
• As this is a new build, this needs testing. The official binaries are CI tested for all releases. Why duplicate testing effort when we already need more testing effort for the official releases?
• openshift-installer releases are tested with a given set of boot images and release images in CI. Any other combination is untested and could lead to unexpected bugs.
• For both oc and openshift-install, you should use a version matching the version of your cluster. This means that packaging will require maintaining several versions in parallel, probably as modules. Note that this also applies to OKD which only has one supported version at a time as your clusters might not all be running the same version.
• OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE and OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE are available for testing and debugging and are otherwise unsupported as a far as I know.

[travier]

We already have Origin packages in Fedora which provide oc.
Origin and its applications are for the 3.x branch of OpenShift, OKD handles 4.x. It also doesn't include openshift-install. Maybe worth submitting for the removal of Origin packages? Currently shipping in F34:

I agree that we should remove all 3.11 packages from Fedora.

If you want to have oc in the Fedora repositories, packaging the latest oc releases (without a rename) would be the easiest path as generally using the latest release with older clusters should work and is the less likely combination to run into issues.

[omenos]

Sorry for the major response delay, been a busy few weeks.

I phrased this as a proposal in this post, but when I brought it up in the WG meeting I was asking more if it was something that other members would consider a good idea and a venture to pursue. This is not a "we should go ahead and do this", I wanted to spark a discussion to see if others viewed this as worthwhile at the current point in time.

I don't have answers for all questions, nor comprehensive ones for the ones I have answered. I was hoping for expanded discussion on what makes this difficult to do presently and potential steps that could mitigate that in the future.

What is the benefit of this packaging?

One obvious benefit is the user doesn't need to browse to grab a tarball, extract it, and manually add things to paths/completions etc. Just a quick dnf install and they get off to work. Essentially lowering barrier of getting started. It's not huge, but something.

Do we have Fedora package request somewhere?

Not to my knowledge. I did a few searches through the fedora-devel list but didn't turn up anything 4.x related. I built these packages the other day just for fun, and decided to look into things afterwords and found this card on the OpenShift/OKD4 project board: https://github.com/orgs/openshift/projects/1#card-28315538

You'll certainly need a new installer to install a different minor version (4.x)

Yes, if the installer were to be packaged it would most likely require the utilization of Modularity to accomplish this, one stream per minor release.

How automated is the new package release procedure?
...
If you want to have oc in the Fedora repositories, packaging the latest oc releases (without a rename) would be the easiest path as generally using the latest release with older clusters should work and is the less likely combination to run into issues.

Right now this is purely POC. So, not at all. The method for both this POC and OpenShift oc is to download the latest released version and run oc version -o yaml to see which commit the build is based off of. Is there an easier way to go about this/a reason why these commits aren't being tagged publicly? In terms of packaging I could see oc as being the baseline target to hit and leave out openshift-install as that's a more complex situation.

Why duplicate testing effort when we already need more testing effort for the official releases?

Excluding the installer, what about the build environments differ between the OKD pipeline and what would be available in Koji to produce RPMs? I know close to nothing about the OKD build process.

[vrutkovs]

That seems to be a valueable addition to OKD, but automation is my main concern. I'm not too familiar with Fedora maintainership, so I'd really prefer to keep the existing process in place and automate Fedora releases as much as possible.

AFAIK Fedora has Anitya, so we can be notified when a new release is uploaded at OKD's github. Can it can also create a new Pagure PR which updates spec files - and I believe it can also trigger Koji builds when merged? Do we need any kind of koji tagging to make it land in bodhi?

[Conan-Kudo]

It's possible with Packit, but not Anitya alone.

[ktdreyer]

As an Openshift user, I want to yum update to get the latest oc binary, so that I do not have to manually update my oc binary by hand.

When we make it easy for users to keep their oc clients up-to-date, we lessen the pressure on OpenShift to support ancient oc clients. It moves the entire ecosystem forward.

[LorbusChris]

I think a good first step would be for a volunteer to create an Fedora package request on BZ, or see if there is an abandoned package that can be revived for this. The request should show how oc (and installer) can be packaged with a proper specfile. Once that's proven (which should'nt be too hard), we can look at automating releases with packit (preferably a new release in https://github.com/openshift/okd would trigger those RPM updates)

[FrostyX]

Hello everyone,
Can we please revisit this discussion again?

COPR mroche/okd

Too bad I wasn't aware of this Copr project when I needed to use the oc command. This is a good start,
if you are concerned that maintaining a package in the official Fedora repositories is too much hassle,
a Copr repository is almost for free.

• We would need to enable fedora-36-x86_64, fedora-37-x86_64, and fedora-rawhide-x86_64
  chroots. There is "Follow Fedora branching" option enabled in the project settings, so when we have
  a rawhide chroot, once there is F38 in the future, the chroot will be automatically enabled in
  the project

• I briefly reviewed the .spec file and it looks good to me. Few improvements:

  • It should install /usr/bin/oc symlink to /usr/bin/okd
  • We should add Provides and Obsoletes fields to indicate that this is a replacement for
    the origin-clients. More documentation here

• We should mention the installation instructions from Copr in the documentation because otherwise,
  nobody won't know about its existence https://docs.okd.io/4.10/cli_reference/openshift_cli/getting-started-cli.html

• You can set up Packit to automatically build the package after every upstream release. See for
  example how this project does it - https://github.com/packit/specfile/blob/main/.packit.yaml#L83 .
  The keywords are job: copr_build and trigger: release. More about Packit here https://packit.dev/docs/guide/

I agree that we should remove all 3.11 packages from Fedora.

If you want to have oc in the Fedora repositories, packaging the
latest oc releases (without a rename) would be the easiest path as
generally using the latest release with older clusters should work
and is the less likely combination to run into issues.

I have nothing against either of those but something needs to be done. Currently, when a user
installs oc from the Fedora repositories via dnf install /usr/bin/oc it installs 3x version which
tracebacks on oc login without any clear
indication of what's wrong. Personally, it took me a couple hours to figure out, that there is
another implementation of oc that we are supposed to use now, and which is not in the Fedora
repositories. That was so outside of the realm of possibilities that it took me hours to even consider.

As this is a new build, this needs testing. The official binaries are
CI tested for all releases. Why duplicate testing effort when we
already need more testing effort for the official releases?

I have no idea how your testing process works, so I can't provide any informed opinion. But since
the official binaries are CI tested, theoretically it should cost us almost nothing to run the same
CI tests (or their subset) for Fedora-packaged binaries. Or there is some blocker?

What are the benefits of packaging for Fedora (or any other
distribution)?

As @omenos, and @ktdreyer already said, it allows users to run dnf install /usr/bin/oc and start
working. We also receive all updates via dnf update, and therefore we don't have to watch the
release cycles of every project, and manually apply updates.

I don't want to be snarky or mean, and I am sorry if it comes across that way, but I just want to
say "How is that even a question?" though.

• Fedora package allows users to install and update the oc tool via package manager.
• Packages in the repositories are signed. Updates spend some time in the testing repositories so
  volunteers can make sure they work. Fedora users trust this process
• The whole point of a Linux distribution is to provide software for its users. We don't install
  packages from PyPI, RubyGems, ..., and from random websites. We install them from Fedora.
• The installation process from documentation
  which says, "download a tarball, and extract a binary from it", is IMHO not safe, and won't be
  trusted by many users. I can't know from what sources and how it was built. How do I know there
  isn't anything malicious inside? I would have to compile from sources, and use my own binary to
  be sure. In comparison, anyone can see the sources used for a Fedora package, see the .spec file
  describing how it is compiled and installed, and anyone can also see the build logs for
  every package in Fedora so we don't have to blindly trust, we can make sure.
• There already is a Homebrew package for macOS. I guess the list of benefits is the same when
  packaging for Fedora
• Many Red Hat employees use Fedora for development, so this change would directly impact them and
  make their life easier
• I want to make clear that I am not saying this last point as a Red Hat employee, this is my
  personal opinion. OpenShift is one of the Red Hat's flagship products and it IMHO throws a bad
  light when its client tools are not even packaged for Fedora. Let other distributions package it
  by themselves, but Fedora is our baby.

How automated is the new package release procedure?

I mentioned how can Packit be used to automatically build packages in Copr. It looks like the setup
for the official Fedora repositories is similar - https://github.com/packit/specfile/blob/main/.packit.yaml#L98

Please let me know if I can help somehow. I can review/sponsor the package, help with the Packit
configuration, whatever you need except for owning the package.

[FrostyX]

Tagging @praiskup, @jpopelka, and @xsuchy who might be interested in reading the discussion.

[LorbusChris]

A good first step would be to get the Packit-as-a-Service GitHub app enabled in the OpenShift org.