📢 Community Catalog will adopt a new format

[camilamacedo86]

📢 Community Catalog will adopt a new format ( from 4.11+ released tags )

The community operator catalog will transition to a new internal format, which is referred to as file-based catalogs. This is mostly a transparent change for users of the Operator Lifecycle Manager. This change impacts you if you use the OPM CLI tool to interact with the index images to filter out operators or add new Operators to it as part of release automation. New commands have been introduced to the OPM utility, which you will need to adopt.

💁 What is a file-based catalog?

Catalogs are responsible for publishing operators, including their entire version history and update graphs. In addition, they make operators installable via the Operator Lifecycle Manager. Today, they are internally based SQLite databases stored in the container image from the content served to the cluster via GRPC APIs to be consumed by OLM. A file-based catalogs is based on a declarative file format that defines the content of a catalog in JSON or YAML. The GRPC interface remains the same.

Cluster administrators and tenants should not see any change to their install workflows and operator consumption with the new format.

Suppose you interact directly with the internal SQLite database or use OPM commands to manipulate the community catalog. In that case, you will need to adopt the new format (see more details below). To know more about this format see the file-based catalog doc.

🙋 Why are we changing the format?

The SQLite format used posed a number of challenges, like as:

• bespoke commands were required to incrementally add new content so that the integrity of the catalog is maintained.
• the imperative nature of adding content to an existing catalog, it was difficult to make changes to existing content in a catalog, for example, adding additional update paths between existing versions or deprecating older versions of an operator.
• the OPM utility could only mutate the state of the catalog it was extremely difficult to recreate a catalogs state from scratch.
• like with all databases, there was a need to back up the catalog state somewhere before making any change so that rollback could occur if needed. In short, it is challenging to maintain an extensive operator catalog in a declarative way without additional external databases and tooling.

Those are some motivations for OLM is moving away from the SQLite database as a storage mechanism. Instead, indexes will serve a plain text-based (JSON or YAML) set of files. The new format allows for a declarative nature of manipulate content, GitOps-friendly, fully automated approach to maintaining operator catalogs.

---

🚀 What does that mean for you?

The conversion to file-based catalogs impacts the following actions:

• build new catalogs from scratch and/or modify and re-publish catalogs using OPM.
• filter catalogs for content via OPM or scripts.
• directly interact with the SQLite database inside the catalog image such as doing SQL queries or running OPM commands.
• prune catalogs for disconnected usage.
• query catalog content outside the cluster.
• debug the index catalog images.

⚠️ If you perform any of the above actions, you will be impacted.

The opm index and opm registry commands that interact with the SQLite database are deprecated as of Openshift 4.9.

This will impact your QE workflow, disconnected customer installs, and CI pipelines that interact with the product catalogs. Make ensure that you read the following sections to know how to deal with the changes.

✨ Otherwise, you are NOT impacted.

You are likely not impacted if you are only using catalog images to:

• mirror them for disconnected clusters using oc adm catalog mirror
• create catalog sources for OLM (to consume the RedHat Index catalog via OLM on the clusters)

Or if you only distribute your Operators by creating a PR in this repository. Nothing will change for you at the first moment. You will still be publishing your projects as you have been using the PR-based contribution process in this project.

If your workflow is not impacted, you can stop reading now and ignore the rest of this document.

NOTE In the future, you will also be able to opt-in to use the file-based catalogs directly and have more flexibility and control to publish your solutions.

⚠️ What this means for OPM users:

OPM users need to shift from an imperative mindset (where their interaction with the index is limited by OPM APIs) to a declarative one (where they can declare their update graph looks like). Instead of starting with an empty catalog image and incrementally adding content with imperative commands like opm index add a YAML file is now the source of truth of what should be in a catalog. OPM will build the catalog reproducible from that file.

A new set of commands has been created for FBC indexes, but these are limited to:

Action	Command
To migrate an SQLite image or DB to the FBC format	Use $ opm migrate
To initialize an updated graph for a package	Use $ opm init
To generate an FBC index from a set of indexes and bundles	Use $ opm render
To prune packages from the index in the FBC format	You can use command lines such as cp and rm -rf to remove to achieve the result of having just a set of the packages. NOTE: opm index prune may be updated in the future to support both formats.
To serve an FBC index	Use$ opm serve
To generate the dockerfile index with the FBC content	Use $ opm alpha generate dockerfile <directory>

The opm index and opm registry commands do not work against an FBC index.

Commands like opm index add are no longer provided as APIs to interact with the index. Instead, FBC users can create their scripts or programs for modifying the FBC in equivalent or custom ways.

Note that the only exception is opm index prune which may be prepared to work with both formats in the future.

💁 What does this means if you:

---

👉 filter catalogs for content via tools or scripts or if you query catalog content

There is no SQLite database anymore to query the content of a catalog. Instead, you can now introspect a YAML/JSON file to verify catalog content.

💡 You can use jq commands or scripts able to work with its JSON/YAML formats. Also, you might would like to use Golang to build the scripts you might want to import the methods provided by operator-registry to work with it. See DeclarativeConfig definition.

HINT OPM will also be defining a set of veneers and APIs that can be compiled to the FBC format. These represent higher-level controls that make adding or changing content in a catalogue more user-friendly, and we will let you know more about it in the future.

👉 debug the images

You might have a better debugging experience where introspecting the index image and understanding what update graph edges and operator versions exist is now more accessible, see:

Command	Description
$ opm render	can be used to programmatically debug a catalog or build other debug tooling against catalogs.
$ opm alpha list	commands can list packages, channels, and bundles in a human-friendly tabular view. These commands are effectively simple wrappers around opm render.

👉 mirror catalogs for disconnected usage with Openshift

For disconnected installs, cluster administrators will need to use:

• commands like cp and rm -rf to reduce the catalogue size to just the set of operators required on the 4.11 tag catalogs to mirror only a subset of operators from catalogs shipped with OpenShift.
• latest versions of OPM and the oc tool for the oc adm catalog mirror command to work on the community operator catalog images with OCP release tag 4.9 or higher.

Note that oc adm catalog mirror command can mirror both SQLite and FBC format catalogs as of oc version 4.9

👉 build new catalogs from scratch and/or modify and re-publish catalogs using OPM

Catalog modifiers (people editing the redhat catalogs) will be impacted depending on how they are doing the modifications. Any workflow based on retrieving and modifying the index.db file, will not be functional for an FBC index.

Consumers of catalogs who don't control the creation of those catalogs should be prepared to handle both SQLite and FBC catalogs. Note that Red Hat will be pivoting its official catalogs to FBC from OpenShift 4.11 release and its tags versions.

Please, check the Creating a file-based catalog image doc to know how to create new indexes.

---

💡 Tips to work with the new FBC format in this transition period or for testing purposes

You can produce the SQL index and then render/migrate the content for the new format. In this case, following some tips:

• use the commands opm registry instead of opm index. (e.g opm registry add [options]).
• note that all options provided by opm index commands are provided via opm registry, less the option to remove the package from the index.
• for file-based catalogs organized by package (like the RedHat Community Index Catalogs), you can use rm -rf to remove the package directory from your file system. OPM does not offer opm registry rm <package>.
• use opm render or opm migrate commands to output the index in your file system in the new format. (e.g.opm migrate registry.redhat.io/redhat/community-operator-index4.9 my-fbc-dir)
• use opm alpha generate dockerfile [options] to generate the index image based on the FBC in your file system and to be able to publish it (e.g. opm alpha generate dockerfile my-fbc-dir)

IMPORTANT: ensure that you will use the latest OPM releases.

---

⏰ When will it happen? What are the tag versions which will be impacted?

Nothing will change for the Openshift tags released up to 4.10.
However, the new format will begin to be used from the 4.11+ tags released.

❓ FAQs:

What are the Openshift/OLM versions which can consume file-based catalog?

The GRPC APIs are backwards compatible so, you can consume file-based catalogs from OpenShift 4.6+.

Where are the catalog images built from this repository?

The content merged in this repository is used to create the following catalogs:

Index Catalog (image)	Description
registry.redhat.io/redhat/community-operator-index	used to publish operators in Openshift. The content provided in this image appears by default in the Openshift and OKD console.

This image can be used with Openshift and OKD as any Kubernetes vendor with OLM installed.

[J0zi]

From contributor point of view, If you are just creating PRs, no change is needed. Index format change will be a change behind the scenes, OLM will behave the same.

[cdjohnson]

For those building catalogs that you intend to consume on OpenShift 4.6-4.9, you MUST embed the CSV and CRDs in the olm.bundle.object in the olm.bundle for the channel heads, or it will not not show up in the OpenShift OperatorHub console nor PackageManifests.

Unless the PackageServer component is updated on all versions of OpenShift, your catalogs simply won't work unless this is done.

https://olm.operatorframework.io/docs/reference/file-based-catalogs/#olmbundleobject-alpha

[camilamacedo86]

🚀**(Reminder) The FBC changes are planned to roll out soon.**

This is a reminder that:

On Feb. 8th: The v4.11 tag of the RedHat Community index image (https://registry.redhat.io/redhat/community-operator-index:v4.11) will be using the file-based catalogs. (All other tags will keep using the existing SQLite format)

[jianzhangbjz]

/cc @bandrade @anpingli

[camilamacedo86]

🚀 The FBC changes rolled out.

From Feb. 9th: The v4.11 tag of the RedHat Community index image (https://registry.redhat.io/redhat/community-operator-index:v4.11) are using the file-based catalogs. (All the other previous tags, < v4.11, will keep using the existing SQLite format)

[nnachefski]

Would it be possible to get an example workflow of pruning the upstream catalog index of all for specified operators? Do all i have to do is replace the 'opm index prune' command with 'rm' commands to get this working agian? The current sqlite workflow i have is:

    echo "- pruning catalog index"
    MIRROR_CATALOG_LINE="./opm index prune -f ${OCP_OPERATORS_UPSTREAM_CATALOG} -p `echo $OCP_OPERATORS_KEEP` -t ${BOOTSTRAP_REGISTRY}/catalog-x86-disconnected:latest"

    echo $MIRROR_CATALOG_LINE
    eval $MIRROR_CATALOG_LINE
    
    echo "- pushing pruned catalog index locally"
    MIRROR_CATALOG_PUSH_LINE="podman push ${BOOTSTRAP_REGISTRY}/catalog-x86-disconnected:latest"
    echo $MIRROR_CATALOG_PUSH_LINE
    eval $MIRROR_CATALOG_PUSH_LINE
    
    echo "- sync'ing catalog locally"
    rm -rf manifests-catalog-x86-disconnected
    MIRROR_CATALOG_IMAGES="oc adm catalog mirror ${BOOTSTRAP_REGISTRY}/catalog-x86-disconnected:latest ${BOOTSTRAP_REGISTRY} --to-manifests=manifests-catalog-x86-disconnected --index-filter-by-os='linux/amd64' --manifests-only -a ${PULL_SECRET_JSON_FILE}"
    echo $MIRROR_CATALOG_IMAGES
    eval $MIRROR_CATALOG_IMAGES
    
    cd manifests-catalog-x86-disconnected
    cat mapping.txt |grep -v -e "^${BOOTSTRAP_REGISTRY}" > mappings.txt
    MIRROR_IMAGES_LINE="oc image mirror --skip-multiple-scopes=true -a ${PULL_SECRET_JSON_FILE} --filter-by-os='linux/amd64' -f mappings.txt"
    echo $MIRROR_IMAGES_LINE
    eval $MIRROR_IMAGES_LINE

A working example would be super helpful in helping folks with 'disconnected' requirements

[camilamacedo86]

Hi @madhukirans,

I raised your question in #793 and we will be working on that. Could you please follow up on this one?

[geoffo-dev]

We're just updating our clusters to v4.11 and just stumbled across this... We use opm index prune as we don't want to move all the images across to a restricted network.

However there is no real easy documentation which explains how to either prune or build an image with redhat operators using this new method that makes the transition easy. The documentation is still based around the sqlite method which now appears is nearly a year out of date...

I think @camilamacedo86 response in #793 does have a method... but it would be better if this could be reflected in the official redhat documentation...