OperatorHub Catalog will adopt a new format

[camilamacedo86]

📢 OperatorHub Catalog will adopt a new format

The K8S Community Operator Catalog will transition to a new internal format, which is referred to as file-based catalogs. This is mostly a transparent change for users of the Operator Lifecycle Manager. This change impacts you if you use the OPM CLI tool to interact with the index images to filter out operators or add new Operators to it as part of release automation. New commands have been introduced to the OPM utility, which you will need to adopt.

💁 What is a file-based catalog?

Catalogs are responsible for publishing operators, including their entire version history and update graphs. In addition, they make operators installable via the Operator Lifecycle Manager. Today, they are internally based SQLite databases stored in the container image from the content served to the cluster via GRPC APIs to be consumed by OLM. A file-based catalogs is based on a declarative file format that defines the content of a catalog in JSON or YAML. The GRPC interface remains the same.

Cluster administrators and tenants should not see any change to their install workflows and operator consumption with the new format.

Suppose you interact directly with the internal SQLite database or use OPM commands to manipulate the community catalog. In that case, you will need to adopt the new format (see more details below). To know more about this format see the file-based catalog doc.

🙋 Why are we changing the format?

The SQLite format used posed a number of challenges, like as:

• bespoke commands were required to incrementally add new content so that the integrity of the catalog is maintained.
• the imperative nature of adding content to an existing catalog, it was difficult to make changes to existing content in a catalog, for example, adding additional update paths between existing versions or deprecating older versions of an operator.
• the OPM utility could only mutate the state of the catalog it was extremely difficult to recreate a catalogs state from scratch.
• like with all databases, there was a need to back up the catalog state somewhere before making any change so that rollback could occur if needed. In short, it is challenging to maintain an extensive operator catalog in a declarative way without additional external databases and tooling.

Those are some motivations for OLM is moving away from the SQLite database as a storage mechanism. Instead, indexes will serve a plain text-based (JSON or YAML) set of files. The new format allows for a declarative nature of manipulate content, GitOps-friendly, fully automated approach to maintaining operator catalogs.

---

🚀 What does that mean for you?

The conversion to file-based catalogs impacts the following actions:

• build new catalogs from scratch and/or modify and re-publish catalogs using OPM.
• filter catalogs for content via OPM or scripts.
• directly interact with the SQLite database inside the catalog image such as doing SQL queries or running OPM commands.
• prune catalogs to reduce the catalogue size to just a set of operators.
• query catalog content outside the cluster.
• debug the index catalog images.

⚠️ If you perform any of the above actions, you will be impacted.

The opm index and opm registry commands that interact with the SQLite database are deprecated as of Openshift 4.9.

This will impact your QE workflow, disconnected customer installs, and CI pipelines that interact with the product catalogs. Make ensure that you read the following sections to know how to deal with the changes.

✨ Otherwise, you are NOT impacted.

You are likely not impacted if you are only using catalog images to create catalog sources for OLM (to consume the OperatotHub Index catalog via OLM on the clusters)

Or if you only distribute your Operators by creating a PR in this repository. Nothing will change for you at the first moment. You will still be publishing your projects as you have been using the PR-based contribution process in this project.

If your workflow is not impacted, you can stop reading now and ignore the rest of this document.

NOTE In the future, you will also be able to opt-in to use the file-based catalogs directly and have more flexibility and control to publish your solutions.

⚠️ What this means for OPM users:

OPM users need to shift from an imperative mindset (where their interaction with the index is limited by OPM APIs) to a declarative one (where they can declare their update graph looks like). Instead of starting with an empty catalog image and incrementally adding content with imperative commands like opm index add a YAML file is now the source of truth of what should be in a catalog. OPM will build the catalog reproducible from that file.

A new set of commands has been created for FBC indexes, but these are limited to:

Action	Command
To migrate an SQLite image or DB to the FBC format	Use $ opm migrate
To initialize an updated graph for a package	Use $ opm init
To generate an FBC index from a set of indexes and bundles	Use $ opm render
To prune packages from the index in the FBC format	You can use command lines such as cp and rm -rf to remove to achieve the result of having just a set of the packages. NOTE: opm index prune may be updated in the future to support both formats.
To serve an FBC index	Use$ opm serve
To generate the dockerfile index with the FBC content	Use $ opm alpha generate dockerfile <directory>

The opm index and opm registry commands do not work against an FBC index.

Commands like opm index add are no longer provided as APIs to interact with the index. Instead, FBC users can create their scripts or programs for modifying the FBC in equivalent or custom ways.

Note that the only exception is opm index prune which may be prepared to work with both formats in the future.

💁 What does this means if you:

---

👉 filter catalogs for content via tools or scripts or if you query catalog content

There is no SQLite database anymore to query the content of a catalog. Instead, you can now introspect a YAML/JSON file to verify catalog content.

💡 You can use jq commands or scripts able to work with its JSON/YAML formats. Also, you might would like to use Golang to build the scripts you might want to import the methods provided by operator-registry to work with it. See DeclarativeConfig definition.

HINT OPM will also be defining a set of veneers and APIs that can be compiled to the FBC format. These represent higher-level controls that make adding or changing content in a catalogue more user-friendly, and we will let you know more about it in the future.

👉 debug the images

You might have a better debugging experience where introspecting the index image and understanding what update graph edges and operator versions exist is now more accessible, see:

Command	Description
$ opm render	can be used to programmatically debug a catalog or build other debug tooling against catalogs.
$ opm alpha list	commands can list packages, channels, and bundles in a human-friendly tabular view. These commands are effectively simple wrappers around opm render.

👉 prune catalogs to reduce the catalogue size to just a set of operators

Commands like cp and rm -rf to reduce the catalogue size to just the set of operators required for the new format tag.

👉 build new catalogs from scratch and/or modify and re-publish catalogs using OPM

Catalog modifiers (people editing the OperatorHub catalogs) will be impacted depending on how they are doing the modifications. Any workflow based on retrieving and modifying the index.db file, will not be functional for an FBC index.

Consumers of catalogs who don't control the creation of those catalogs should be prepared to handle both SQLite and FBC catalogs. Note that OperatorHub will be pivoting its catalogs to FBC (Check the section When will it happen? What are the timelines?).

Please, check the Creating a Catalog doc to know how to create new indexes.

---

💡 Tips to work with the new FBC format in this transition period or for testing purposes

You can produce the SQL index and then render/migrate the content for the new format. In this case, following some tips:

• use the commands opm registry instead of opm index. (e.g opm registry add [options]).
• note that all options provided by opm index commands are provided via opm registry, less the option to remove the package from the index.
• for file-based catalogs organized by package (like the OperatorHub Index Catalogs), you can use rm -rf to remove the package directory from your file system. OPM does not offer opm registry rm <package>.
• use opm render or opm migrate commands to output the index in your file system in the new format. (e.g.opm migrate quay.io/operatorhubio/catalog my-fbc-dir)
• use opm alpha generate dockerfile [options] to generate the index image based on the FBC in your file system and to be able to publish it (e.g. opm alpha generate dockerfile my-fbc-dir)

IMPORTANT: ensure that you will use the latest OPM releases.

---

🚀 How can I start to consume the new format?

To help you out be adapted, a temporary tag tmp_latest_fbc was created. You can use it to check if you or your workflows will be impacted as to working on the required changes.

⚠️ it is planned to remove quay.io/operatorhubio/catalog:tmp_latest_fbc tag a few weeks after the default quay.io/operatorhubio/catalog:latest began to be published with the new format, FBC.

🙇 I cannot start to consume the new format. I need more time to be adapted. What can I do?

If you know that will be impacted and cannot address the required changes:

You can replace the latest tag with the new temporary tmp_latest_sql tag in your workflows to avoid any impact.

The quay.io/operatorhubio/catalog:tmp_latest_sql contains the same content with the SQLite database storage mechanism.

⚠️ Please, you must be aware that the quay.io/operatorhubio/catalog:tmp_latest_sql tag will be provided up Apr/2022. It is planned to be removed.

⏰ When will it happen? What are the timelines?

• ✨ Dez/2022: New tags released

Image Tag	Format	⚠️ Attention
quay.io/operatorhubio/catalog:tmp_latest_fbc	FBC	👷 Use this temporary tag to adapt your workflows to the new format. Be aware that it will be removed at Mar/2022.
quay.io/operatorhubio/catalog:tmp_latest_sql	SQLite	🙆‍♀️ Use this tag to not be impacted by the changes and up your ability to consume the new format, FBC. This tag will be removed at Apr/2022.
quay.io/operatorhubio/catalog:latest	SQLite	✨ Nothing change up Feb/2022.

• 🔥 Feb/2022: the quay.io/operatorhubio/catalog:latest tag will begin to be provided with FBC format.
• ⚠️ Mar/2022: the temporary quay.io/operatorhubio/catalog:tmp_latest_fbc tag created to help you out will be removed.
• ⚠️ Apr/2022: the temporary quay.io/operatorhubio/catalog:tmp_latest_sql tag created to help you out will be removed.

---

💡 How can I migrate an Index catalog from SQLite to FBC format locally and test it out?

Following the steps using the quay.io/operatorhubio/catalog:tmp_latest_sql index as an example.

1. Create a new directory:

$ mkdir operatorhub

2. Use opm migrate or render commands to convert the image format:

$ opm migrate quay.io/operatorhubio/catalog:tmp_latest_sql operatorhubio

Now, you can check the image content in the directory created using the FBC format.

3. Generate an index with the FBC content and publish it in your registry:

$ opm alpha generate dockerfile operatorhubio
$ docker build -t <your-registry>/operatorhub:fbc -f operatorhubio.Dockerfile .
$ docker push <your-registry>/operatorhub:fbc 

Now, you can use this image to check the impact in your workflows or to just check its consumption in a cluster by creating a CatalogSource to verify it (More info).

You might want to check Creating an update graph with OLM to know more about how the upgrade graph works with the new format.

[camilamacedo86]

🚀**(Reminder) The FBC changes are planned to roll out soon.**

This is a reminder that:
On Feb. 1st: The latest tag of OperatorHub catalog index image (quay.io/operatorhubio/catalog:latest) will be using the file-based catalogs format.

[mvalarh]

quay.io/operatorhubio/catalog:latest (k8s upstream index) is in FBC (file-based catalogs format) from now on

[camilamacedo86]

🚀 The FBC changes rolled out.

From Feb. 1st: The latest tag of OperatorHub catalog index image (quay.io/operatorhubio/catalog:latest) are using the file-based catalogs format.