diff --git a/knowledge/oc-mirror/qna.yaml b/knowledge/oc-mirror/qna.yaml new file mode 100644 index 000000000..e7bdb605f --- /dev/null +++ b/knowledge/oc-mirror/qna.yaml @@ -0,0 +1,154 @@ +created_by: CID +domain: OpenShift +seed_examples: + - question: what is oc-mirror? + answer: | + You can use the oc-mirror OpenShift CLI (oc) plugin to mirror all required OpenShift Container Platform content and other images to your mirror registry by using a single tool. It provides the following features: + * Provides a centralized method to mirror OpenShift Container Platform releases, Operators, helm charts, and other images. + * Maintains update paths for OpenShift Container Platform and Operators. + * Uses a declarative image set configuration file to include only the OpenShift Container Platform releases, Operators, and images that your cluster needs. + * Performs incremental mirroring, which reduces the size of future image sets. + * Prunes images from the target mirror registry that were excluded from the image set configuration since the previous execution. + * Optionally generates supporting artifacts for OpenShift Update Service (OSUS) usage. + - question: Can we do a --dry-run before doing the actual mirror ? + answer: | + Yes, --dry-run option is available for v2 as well and command to perform the same is `oc-mirror --config file://dryrun --dry-run`. When running dry-run on a mirror to disk workflow, oc-mirror v2 also checks if all the images within the image set are available in its cache. Any missing images are added to the missing.txt file. + When a dry-run is performed before mirroring both missing.txt and mapping.txt will contain the same list of images.. Adding the sample output here + - question: What can be mirrored with oc-mirror ? + answer: | + ocp platform release images(only stable ones), operator catalogs(again only stable ones), oci based catalogs, additionalImages + BlockedImages functionality is only available for images with manifest unknown errors which means if an image mirroring is failing due to `manifestUnknown` that image could be added to blockedImages and image will be ignored for mirroring. + Helm charts cannot be mirrored and are not available for now. + - question: how do you create an imagesetconfiguration? + answer: | + Use the oc mirror init command to create a template for the image set configuration and save it to a file called imageset-config.yaml: + $ oc mirror init <--registry > imageset-config.yaml + Specifies the location of your storage backend, such as example.com/mirror/oc-mirror-metadata. + Edit the file and adjust the settings as necessar + Edit the file and adjust the settings as necessary: + + kind: ImageSetConfiguration + apiVersion: mirror.openshift.io/v1alpha2 + archiveSize: 4 + storageConfig: + registry: + imageURL: example.com/mirror/oc-mirror-metadata + skipTLS: false + mirror: + platform: + channels: + - name: stable-4.17 + type: ocp + graph: true + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 + packages: + - name: serverless-operator + channels: + - name: stable + additionalImages: + - name: registry.redhat.io/ubi9/ubi:latest + helm: {} + + Add archiveSize to set the maximum size, in GiB, of each file within the image set. + Set the back-end location to save the image set metadata to. This location can be a registry or local directory. It is required to specify storageConfig values. + Set the registry URL for the storage backend. + Set the channel to retrieve the OpenShift Container Platform images from. + Add graph: true to build and push the graph-data image to the mirror registry. The graph-data image is required to create OpenShift Update Service (OSUS). The graph: true field also generates the UpdateService custom resource manifest. The oc command-line interface (CLI) can use the UpdateService custom resource manifest to create OSUS. For more information, see About the OpenShift Update Service. + Set the Operator catalog to retrieve the OpenShift Container Platform images from. + Specify only certain Operator packages to include in the image set. Remove this field to retrieve all packages in the catalog. + Specify only certain channels of the Operator packages to include in the image set. You must always include the default channel for the Operator package even if you do not use the bundles in that channel. You can find the default channel by running the following command: oc mirror list operators --catalog= --package=. + Specify any additional images to include in image set. + - question: what is oc-mirror used for? + answer: you can also use oc-mirror plugin to mirror images to a mirror registry in your fully or partially disconnected environments. + - question: how do you install oc-mirror? + answer: Navigate to the Downloads page of the OpenShift Cluster Manager. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file. Extract the archive by `$ tar xvzf oc-mirror.tar.gz`. If necessary, update the plugin file to be executable `$ chmod +x oc-mirror`` + - question: what is a valid imagesetconfiguration? + answer: | + kind: ImageSetConfiguration + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + platform: + channels: + - name: stable-4.13 + minVersion: 4.13.10 + maxVersion: 4.13.10 + graph: true + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 + packages: + - name: aws-load-balancer-operator + - name: 3scale-operator + - name: node-observability-operator + additionalImages: + - name: registry.redhat.io/ubi8/ubi:latest + - name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0 + - question: how do I use oc-mirror to mirror to a partially disconnected environment? + answer: | + In a partially disconnected environment, you can mirror an image set directly to the target mirror registry + you can use the following command line oc mirror -c isc.yaml --workspace file:// docker:// --v2 + You can use the oc-mirror plugin to mirror an image set directly to a target mirror registry that is accessible during image set creation. + You are required to specify a storage backend in the image set configuration file. This storage backend can be a local directory or a Docker v2 registry. The oc-mirror plugin stores metadata in this storage backend during image set creation. + Do not delete or modify the metadata that is generated by the oc-mirror plugin. You must use the same storage backend every time you run the oc-mirror plugin for the same mirror registry. + Prerequisites + * You have access to the internet to get the necessary container images. + * You have installed the OpenShift CLI (oc). + * You have installed the oc-mirror CLI plugin. + * You have created the image set configuration file. + Procedure + Run the oc mirror command to mirror the images from the specified image set configuration to a specified registry: + $ oc mirror --config=./ \ + docker://registry.example:5000 + Specify the image set configuration file that you created. For example, imageset-config.yaml. + Specify the registry to mirror the image set file to. The registry must start with docker://. If you specify a top-level namespace for the mirror registry, you must also use this same namespace on subsequent executions. + - question: What do I do after I finish mirroring an imageset with oc-mirror? + answer: | + Navigate into the oc-mirror-workspace directory that was generated. + Navigate into the results directory, for example, results-1639608409/. + Verify that YAML files are present for the ImageContentSourcePolicy and CatalogSource resources. + The repositoryDigestMirrors section of the ImageContentSourcePolicy YAML file is used for the install-config.yaml file during installation. + Configure your cluster to use the resources generated by oc-mirror + After you have mirrored your image set to the mirror registry, you must apply the generated ImageContentSourcePolicy, CatalogSource, and release image signature resources into the cluster. + The ImageContentSourcePolicy resource associates the mirror registry with the source registry and redirects image pull requests from the online registries to the mirror registry. The CatalogSource resource is used by Operator Lifecycle Manager (OLM) to retrieve information about the available Operators in the mirror registry. The release image signatures are used to verify the mirrored release images. + * Log in to the OpenShift CLI as a user with the cluster-admin role. + * Apply the YAML files from the results directory to the cluster by running the following command + oc apply -f ./oc-mirror-workspace/results-1639608409/ + * If you mirrored release images, apply the release image signatures to the cluster by running the following command: + oc apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/ + - question: how do I use oc-mirror to create an archive containing all the images needed for a mirroring? + answer: you can use the following command line oc mirror -c isc.yaml file:// --v2 + - question: How can i discover content to mirror using oc-mirror for releases and operator images ? + answer: | + oc-mirror provides a way to discover OpenShift release and operator content, then use that information to craft mirror payloads. The list updates command traverses update graphs between the last oc mirror run and provided configuration to show what new versions are available. + * Updates + List updates since the last oc-mirror run + oc-mirror list updates --config imageset-config.yaml + Note: You must have existing metadata in your workspace (or remote storage, if using) to use list updates + * Releases + ** List all available release payloads for a version of OpenShift in the stable channel (the default channel) + *** oc-mirror list releases --version=4.9 + ** List all available release channels to query for a version of OpenShift + ***oc-mirror list releases --channels --version=4.9 + ** List all available release payloads for a version of OpenShift in a specific channel + *** oc-mirror list releases --channel=fast-4.9 + * Operators + ** List all available Operator catalogs for a version of OpenShift + *** oc-mirror list operators --catalogs --version=4.9 + ** List all available Operator packages in a catalog + *** oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.9 + ** List all available Operator channels in a package + *** oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.9 --package=kiali + ** List all available Operator versions in a channel + *** oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.9 --package=kiali --channel=stable +task_description: "" +document: + repo: https://github.com/openshift/oc-mirror + commit: 95f0611c1dc9584a4a9e857912b9eaa539234bbc + patterns: + - docs/usage.md + - docs/overview.md + - v2/docs/delete-functionality.md + - v2/docs/enclave_support.md + + + diff --git a/knowledge/oc-mirror/qna2.yaml b/knowledge/oc-mirror/qna2.yaml new file mode 100644 index 000000000..d79246a34 --- /dev/null +++ b/knowledge/oc-mirror/qna2.yaml @@ -0,0 +1,400 @@ +--- +version: 3 +created_by: CID +domain: OpenShift +seed_examples: + - context: | + **oc-mirrorv2** This section mostly talks about what is oc-mirrorv2, where to download the bits and how to invoke it. + to start with oc-mirror v2. + questions_and_answers: + - question: | + why oc-mirror v2 ? + answer: | + oc-mirror v2 has been developed to address challenges faced in v1, + ensure seamless experience for end users while using oc-mirror and + also one of the use cases that we are trying to address here is + enclave support. Currently this is in TechPreview for `4.16` and + current plans of today is to GA in a later version + - question: | + Where can I download oc-mirror v2 bits ? + answer: | + There is no separate binary for oc-mirror v2, it is the same as v1 and + you could follow the same process to download like v1 + - question: | + How do I invoke oc-mirror v2 ? + answer: | + users/customers trying/wanting to work with oc-mirror v2 will need to + download the latest oc-mirror bits the same way they would do for + v1 and when running the command add `--v2` which will invoke + oc-mirror v2 for you. + - context: | + **oc-mirrorv2** List all commands, options available with v2, generate a sample ISC and --dry-run + questions_and_answers: + - question: | + List all commands and options available for oc-mirror v2 + answer: > + To list all commands and options that are available for oc-mirror v2 use + + command `oc-mirror --v2 -h` + - question: > + Is there a command to generate sample ImageSetConfig for oc-mirror v2 ? + answer: > + Currently we do not have a way to generate ImageSetConfig like we have + + for v1 which is `oc-mirror --init`. However the plan is to have this + + feature included by v4.17. In the meanwhile you could use the + + following ImageSetConfig as a sample one + kind: ImageSetConfiguration + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + platform: + channels: + - name: stable-4.15 + type: ocp + graph: true + operators: + - catalog: oci:///home/knarra/ocmirrortest/ibm-catalog + targetTag: v14 + targetCatalog: knarra/catalog + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 + packages: + - name: advanced-cluster-management + additionalImages: + - name: registry.redhat.io/ubi8/ubi:latest + - name: quay.io/openshifttest/hello-openshift@sha256:4200f438cf2e9446f6bcff9d67ceea1f69ed07a2f83363b7fb52529f7ddd8a83 + - question: | + Can we do a --dry-run before doing the actual mirror ? + answer: > + Yes, --dry-run option is available for v2 as well and command to perform + + the same is `oc-mirror --config file://dryrun + + --dry-run`. When running dry-run on a mirror to disk workflow, + + oc-mirror v2 also checks if all the images within the image set are + + available in its cache. Any missing images are added to the + + missing.txt file. + + + When a dry-run is performed before mirroring both missing.txt and mapping.txt will contain the same list of images.. Adding the sample output here + + + [knarra@knarra-thinkpadx1carbon7th ocmirrortest]$ oc-mirror --config /tmp/isc_dryrun.yaml file://CLID-19 --dry-run --v2 + + + 2024/04/25 15:28:21 [INFO] : :warning: --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. + + + 2024/04/25 15:28:21 [INFO] : :wave: Hello, welcome to oc-mirror + + + 2024/04/25 15:28:21 [INFO] : :gear: setting up the environment for you... + + + 2024/04/25 15:28:21 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk + + + 2024/04/25 15:28:21 [INFO] : :sleuth_or_spy: going to discover the necessary images... + + + 2024/04/25 15:28:21 [INFO] : :mag: collecting release images... + + + 2024/04/25 15:28:21 [INFO] : :mag: collecting operator images... + + + 2024/04/25 15:28:31 [INFO] : :mag: collecting additional images... + + + 2024/04/25 15:28:31 [WARN] : :warning: 54/54 images necessary for mirroring are not available in the cache. + + + 2024/04/25 15:28:31 [WARN] : List of missing images in : CLID-19/working-dir/dry-run/missing.txt. + + + please re-run the mirror to disk process + + + 2024/04/25 15:28:31 [INFO] : :page_facing_up: list of all images for mirroring in : CLID-19/working-dir/dry-run/mapping.txt + + + 2024/04/25 15:28:31 [INFO] : mirror time : 9.641091076s + + + 2024/04/25 15:28:31 [INFO] : :wave: Goodbye, thank you for using oc-mirror + - context: | + **oc-mirrorv2** oc-mirror v2 capabilities, new changes that customers would see while using v2 and content discovery, changes in ISC + questions_and_answers: + - question: | + What can be mirrored with --v2 ? + answer: > + ocp platform release images(only stable ones), operator catalogs(again + + only stable ones), oci based catalogs, additionalImages + + + BlockedImages functionality is only available for images with manifest unknown errors which means if an image mirroring is failing due to `manifestUnknown` that image could be added to blockedImages and image will be ignored for mirroring. Helm charts cannot be mirrored and are not available for now. + - question: > + What is new with oc-mirror v2 that customers will experience when + working with it ? + answer: > + Discovery of content is not possible + + + * Unlike v1, v2 cannot do content discovery, however users could still do the content discovery using v1 commands and it is still valid for v2, Command like oc-mirror list releases; oc-mirror list operators + + + No oc-mirror-workspace + + + * In oc-mirror v1 when custom resources like `idms/catalogsource/updateService` files are generated they are generated under oc-mirror-workspace whereas in v2 we have a new folder called `working-dir/cluster-resources` and all resources are generated under this. + + + No rebuild of catalogs + + + * Catalogs will not be rebuilt in v2 unlike v1 which means when a user/customer tries to mirror an operator and its specific version he/she will see all the operators from that catalog in the Openshift console irrespective of what was mirrored. Only metadata of other operators that are not mentioned in the imageSetConfig will be mirrored so user/customer does not need to worry about the mirroring time nor about the registry disk space + + + IDMS & ITMS Generation + + + * IDMS, ITMS, catalog source & UpdateService will be generated under the workspace provided. + + + * All images by tag are added to ITMS while all images by digest are added to IDMS. + + + * One ITMS and IDMS is generated, each file will contain several custom resources one per category of images mirrored. + + + * Unlike v1 user/customer does not need to merge the IDMS files generated and these could be applied cumulatively, just make sure to update the metadata.name so that it does not throw `Already exists error` + + + * Introduces catalogSourceTemplate to imageSetConfig in order to allow the user customization of the catalogSource, if provided will add updateStrategy to the generated catalogSource. Below blocks how to set this option in the ImageSetConfig file and also how catalogSource gets generated when it is set. + + + kind: ImageSetConfiguration + + + apiVersion: mirror.openshift.io/v2alpha1 + + + archiveSize: 1 + + + mirror: + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14 + packages: + - name: aws-load-balancer-operator + targetCatalogSourceTemplate: "/home/skhoury/go/src/github.com/openshift/oc-mirror/v2/tests/catalog-source_template.yaml" + + kind: ImagesetConfiguration + + + apiVersion: operators.coreos.com/v1alpha1 + + kind: CatalogSource + + metadata: + name: totalRubbish + namespace: openshift-marketplace + spec: + image: totalRubbish + sourceType: grpc + updateStrategy: + registryPoll: + interval: 30m0s + Filter specific bundles of catalog images + + + * Using min and max versions requires lots of disk space as it mirrors all intermediate versions as well, so with v2 users/customers can now specify the exact bundles of an operator that they would like to mirror. + + + * Currently there is no official tool that supports how to filter the bundles of an operator, however one could use the opm tool to discover the content as shown below + + + * opm alpha list bundles registry.redhat.io/redhat/community-operator-index:v4.12 etcd + + + PACKAGE CHANNEL BUNDLE REPLACES SKIPS SKIP RANGE IMAGE + + etcd alpha etcdoperator.v0.6.1 quay.io/openshift-community-operators/etcd@sha256:e42581eacbc8349f40c6a4ed6d7d82d6168518e15b1a24cba36817205bd8688f + etcd clusterwide-alpha etcdoperator.v0.9.0 quay.io/openshift-community-operators/etcd@sha256:07157e50458e879c10975e580445f2b378947edc6c918ea873223ef3ba480218 + etcd clusterwide-alpha etcdoperator.v0.9.2-clusterwide etcdoperator.v0.9.0 quay.io/openshift-community-operators/etcd@sha256:23d47faa666e614bd4f2c1fd4f04a5ce69cd61e0dde58843ffd9f8cf642601f6 + etcd clusterwide-alpha etcdoperator.v0.9.4-clusterwide etcdoperator.v0.9.2-clusterwide + * Example below shows imageSetConfig of mirroring using specific bundles of an operator + kind: ImagesetConfiguration + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14 + packages: + - name: aws-load-balancer-operator + bundles: + - name: aws-load-balancer-operator.v1.1.0 + - name: aws-load-balancer-operator.v1.1.1 + - name: 3scale-operator + bundles: + - name: 3scale-operator.v0.10.0-mas + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 + packages: + - name: aws-load-balancer-operator + bundles: + - name: aws-load-balancer-operator.v0.0.1 + - name: aws-load-balancer-operator.v0.2.0 + - name: 3scale-operator + bundles: + - name: 3scale-operator.v0.10.5 + - name: 3scale-operator.v0.11.11 + * If the operator could not be found for some reason oc-mirror will skip and continue and does not end up failing + * Mixing both filtering by bundles and filtering by channels or minVersion/maxVersion is not allowed + Support Incremental Mirroring using date + * If --since 2024-03-28 is specified oc-mirror v2 will mirror content from this date, if not will mirror everything new from the previously mirrored content. + * `oc-mirror --v2 -c clid-9.yaml file:///home/ocmirroruser/usertest/ --since 2024-01-01` + TargetTag And TargetCatalog + * There is a slight change in targetCatalog and targetTag works in v2. These are not mutually exclusive + * Listing valid & invalid examples below + * Invalid Example + kind: ImageSetConfiguration + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14 + targetCatalog: abc/def:v5.5 + packages: + - name: aws-load-balancer-operator + * Valid Example + kind: ImageSetConfiguration + apiVersion: mirror.openshift.io/v2alpha1 + mirror: + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14 + targetCatalog: abc/def + targetTag: v4.4 + packages: + - name: aws-load-balancer-operator + No Auto-pruning of images + * If a customer/user would like to delete some operator/images from the registry they will need to generate a delete-images.yaml file and use that yaml file to delete images from the registry. When a user deletes the images only manifests will be deleted from the registry and user/admin needs to run garbage collect on the registry to completely remove it from the registry. Manifests that are common across the releases or operators will not be deleted and only images that are untagged will be removed from the registry when garbage collect is run on the registry. + * User will need to create a ImageSetConfig.yaml file with kind DeleteImageSetConfiguration with all the images he would like to delete. Example as shown below + apiVersion: mirror.openshift.io/v2alpha1 + kind: DeleteImageSetConfiguration + delete: + platform: + channels: + - name: stable-4.13 + minVersion: 4.13.3 + maxVersion: 4.13.3 + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13 + packages: + - name: 3scale-operator + additionalImages: + - name: registry.redhat.io/ubi8/ubi-minimal:latest + * oc-mirror delete --config delete-isc.yaml --workspace file://clid-20 --v2 --generate -> Generates working-dir/delete/delete-images.yaml but does not delete anything. + * oc-mirror delete --destination docker://localhost:5000/clid-20 --v2 --delete-yaml-file working-dir/delete/delete-images.yaml -> Delete images (Untagging) + * Run garbage collection at the registry end to completely remove the images. + * When `--force-cache-delete=true` is set and also you have done the full MirrorToDisk and DiskToMirror that is the only time it will delete the cache as well along with the images in registry + * In the mirrorTomirror workflows there is no cache so even if you use `--force-cache-delete=true` it will ignore it + * Note: should try avoid deleting the cache", only as a last resort if disk space is a premium, and before doing so making a backup of all tar.gz as well as the current cache + - question: | + what changes do we see in the ImageSetConfig from v1 to v2 + answer: | + Yes, the storageConfig option is completely removed and we do not use + that any more. If added it will be ignored, instead it uses local + cache to serve images more efficiently. Also, the api version is + v2alpha1 + - context: | + **oc-mirrorv2** IDMS, ITMS, errors during mirroring & how to troubleshoot + questions_and_answers: + - question: | + Do we generate ICSP and CatalogSource after mirroring ? + answer: > + ISCP is no longer supported . so IDMS, ITMS & catalog source is + generated. + - question: | + How to review errors that happened during mirroring ? + answer: | + If mirroring fails, all the errors are displayed on the stdout and at + the end oc-mirror v2 generates a text file under working-dir with name + `mirroring_errors__