feat: Replace base image with RHEL UBI

[flamarion]

This change could unblock the Operator for many customers in OpenShift. Some customers in special, require the Operator built on top of RHEL UBI, which is the nature of this change.
Based on preliminary tests, this change will not impact the way the Operator is executed but will, at the same time, fulfill the requirements described.
On top of it, put us one step ahead and start thinking about getting the W&B Operator certified by Red Hat.

Build:

make docker-build
test -s /Users/flamarion/labs/operator-rhel-ubi/operator/bin/controller-gen && /Users/flamarion/labs/operator-rhel-ubi/operator/bin/controller-gen --version | grep -q v0.14.0 || \
        GOBIN=/Users/flamarion/labs/operator-rhel-ubi/operator/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
/Users/flamarion/labs/operator-rhel-ubi/operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/flamarion/labs/operator-rhel-ubi/operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
test -s /Users/flamarion/labs/operator-rhel-ubi/operator/bin/setup-envtest || GOBIN=/Users/flamarion/labs/operator-rhel-ubi/operator/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
KUBEBUILDER_ASSETS="/Users/flamarion/labs/operator-rhel-ubi/operator/bin/k8s/1.26.0-darwin-arm64" go test ./... -coverprofile cover.out
        github.com/wandb/operator/api/v1                coverage: 0.0% of statements
        github.com/wandb/operator               coverage: 0.0% of statements
?       github.com/wandb/operator/controllers/internal/v1       [no test files]
?       github.com/wandb/operator/pkg/redact    [no test files]
        github.com/wandb/operator/pkg/helm              coverage: 0.0% of statements
        github.com/wandb/operator/pkg/wandb/spec/charts         coverage: 0.0% of statements
        github.com/wandb/operator/pkg/utils             coverage: 0.0% of statements
        github.com/wandb/operator/pkg/wandb/spec/channel/deployer/deployerfakes         coverage: 0.0% of statements
        github.com/wandb/operator/pkg/wandb/spec/specfakes              coverage: 0.0% of statements
        github.com/wandb/operator/pkg/wandb/spec/state/statefakes               coverage: 0.0% of statements
        github.com/wandb/operator/pkg/wandb/status              coverage: 0.0% of statements
ok      github.com/wandb/operator/controllers   8.815s  coverage: 54.2% of statements
ok      github.com/wandb/operator/controllers/internal/ctrlqueue        1.982s  coverage: 71.4% of statements
ok      github.com/wandb/operator/pkg/wandb/spec        0.859s  coverage: 0.0% of statements
ok      github.com/wandb/operator/pkg/wandb/spec/channel/deployer       12.687s coverage: 86.5% of statements
ok      github.com/wandb/operator/pkg/wandb/spec/operator       1.345s  coverage: 100.0% of statements
ok      github.com/wandb/operator/pkg/wandb/spec/state  0.647s  coverage: 100.0% of statements
ok      github.com/wandb/operator/pkg/wandb/spec/state/secrets  6.131s  coverage: 76.0% of statements
ok      github.com/wandb/operator/pkg/wandb/spec/utils  2.053s  coverage: 77.8% of statements
docker build -t wandb/controller:latest .
[+] Building 1.1s (21/21) FINISHED                                                                                                                                                                               docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                                             0.0s
 => => transferring dockerfile: 1.63kB                                                                                                                                                                                           0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi9/ubi:latest                                                                                                                                                      0.5s
 => [internal] load metadata for docker.io/library/golang:1.23                                                                                                                                                                   1.0s
 => [auth] library/golang:pull token for registry-1.docker.io                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                0.0s
 => => transferring context: 241B                                                                                                                                                                                                0.0s
 => [manager-builder  1/11] FROM docker.io/library/golang:1.23@sha256:574185e5c6b9d09873f455a7c205ea0514bfd99738c5dc7750196403a44ed4b7                                                                                           0.0s
 => [internal] load build context                                                                                                                                                                                                0.0s
 => => transferring context: 9.29kB                                                                                                                                                                                              0.0s
 => [stage-1 1/3] FROM registry.access.redhat.com/ubi9/ubi:latest@sha256:1057dab827c782abcfb9bda0c3900c0966b5066e671d54976a7bcb3a2d1a5e53                                                                                        0.0s
 => CACHED [manager-builder  2/11] WORKDIR /workspace                                                                                                                                                                            0.0s
 => CACHED [manager-builder  3/11] COPY go.mod go.mod                                                                                                                                                                            0.0s
 => CACHED [manager-builder  4/11] COPY go.sum go.sum                                                                                                                                                                            0.0s
 => CACHED [manager-builder  5/11] RUN go mod download                                                                                                                                                                           0.0s
 => CACHED [manager-builder  6/11] COPY main.go main.go                                                                                                                                                                          0.0s
 => CACHED [manager-builder  7/11] COPY api/ api/                                                                                                                                                                                0.0s
 => CACHED [manager-builder  8/11] COPY pkg/ pkg/                                                                                                                                                                                0.0s
 => CACHED [manager-builder  9/11] COPY controllers/ controllers/                                                                                                                                                                0.0s
 => CACHED [manager-builder 10/11] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go                                                                                                                      0.0s
 => CACHED [manager-builder 11/11] RUN mkdir -p /helm/.cache/helm /helm/.config/helm /helm/.local/share/helm && chmod -R 1777 /helm                                                                                              0.0s
 => CACHED [stage-1 2/3] COPY --from=manager-builder /workspace/manager .                                                                                                                                                        0.0s
 => CACHED [stage-1 3/3] COPY --from=manager-builder /helm /helm                                                                                                                                                                 0.0s
 => exporting to image                                                                                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                                                                                          0.0s
 => => writing image sha256:d248a618e529dbbd73648106a349b544a9ddd28989e1a77f9d0bf067ee8283a4                                                                                                                                     0.0s
 => => naming to docker.io/wandb/controller:latest

Running the operator locally with the new image

docker run --rm -e KUBECONFIG=$KUBECONFIG --mount type=bind,source=/Users/flamarion/labs/kubernetes-home/config,target=/Users/flamarion/labs/kubernetes-home/config,readonly wandb/controller:latest
2024-12-06T14:06:46Z    INFO    setup   starting manager
2024-12-06T14:06:46Z    INFO    controller-runtime.metrics      Starting metrics server
2024-12-06T14:06:46Z    INFO    starting server {"name": "health probe", "addr": "[::]:8081"}
2024-12-06T14:06:46Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-12-06T14:06:46Z    INFO    Starting EventSource    {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "source": "kind source: *v1.WeightsAndBiases"}
2024-12-06T14:06:46Z    INFO    Starting EventSource    {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "source": "kind source: *v1.Secret"}
2024-12-06T14:06:46Z    INFO    Starting EventSource    {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "source": "kind source: *v1.ConfigMap"}
2024-12-06T14:06:46Z    INFO    Starting Controller     {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases"}
2024-12-06T14:06:46Z    INFO    Starting workers        {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "worker count": 1}
2024-12-06T14:06:46Z    INFO    === Reconciling Weights & Biases instance...    {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "WeightsAndBiases": {"name":"wandb","namespace":"default"}, "namespace": "default", "name": "wandb", "reconcileID": "91025dd8-d0ea-4f04-ae1a-308a7fcdda08", "NamespacedName": {"name":"wandb","namespace":"default"}, "Name": "wandb", "start": true}
2024-12-06T14:06:47Z    INFO    Found Weights & Biases instance, processing the spec... {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "WeightsAndBiases": {"name":"wandb","namespace":"default"}, "namespace": "default", "name": "wandb", "reconcileID": "91025dd8-d0ea-4f04-ae1a-308a7fcdda08", "Spec": {"chart":{},"values":{}}}
2024-12-06T14:06:47Z    DEBUG   events  Reconciling     {"type": "Normal", "object": {"kind":"WeightsAndBiases","namespace":"default","name":"wandb","uid":"373b23ee-1842-4a69-ad8d-0f0abfe2bdb3","apiVersion":"apps.wandb.com/v1","resourceVersion":"112561711"}, "reason": "Reconciling"}
2024-12-06T14:06:47Z    DEBUG   events  Loading desired configuration   {"type": "Normal", "object": {"kind":"WeightsAndBiases","namespace":"default","name":"wandb","uid":"373b23ee-1842-4a69-ad8d-0f0abfe2bdb3","apiVersion":"apps.wandb.com/v1","resourceVersion":"112561711"}, "reason": "LoadingConfig"}
2024-12-06T14:06:47Z    INFO    Desired spec    {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "WeightsAndBiases": {"name":"wandb","namespace":"default"}, "namespace": "default", "name": "wandb", "reconcileID": "91025dd8-d0ea-4f04-ae1a-308a7fcdda08", "spec": {"metadata":{"channelId":"f0081406-b0a7-4c51-9c2d-1b911aa68b09","channelInheritsFrom":"b56e1972-3c78-4de0-af90-e3597bb0785a","channelName":"OpenShift - Flamarion","releaseCreatedAt":"2024-12-06T00:52:10.678Z","releaseExtends":"6368f36f-5854-445f-a460-f8ea3fc6d799","releaseId":"e47c97cf-53b9-4965-9246-9e0aa32831df","releaseName":"v202412-5.16"},"chart":{"url":"https://charts.wandb.ai","name":"operator-wandb","version":"0.18.17","password":"","username":""},"values":{"app":{"image":{"repository":"wandb/local","tag":"0.63.0"}},"console":{"image":{"repository":"wandb/console","tag":"2.13.1"}},"flat-run-fields-updater":{"image":{"tag":"0.63.0"}},"global":{"bucket":{"accessKey":"***","name":"pves3.home.lab","path":"wandb","provider":"s3","region":"default","secretKey":"wsnNA1Vq2RHbKdrXSTaw0a09h79QQBZk0AXRUMNY"},"extraEnv":{"ENABLE_REGISTRY_UI":true,"GORILLA_ACTIVITY_STORE_BACKFILL_ENABLE":"true","GORILLA_ACTIVITY_STORE_ENABLE":"true","GORILLA_ACTIVITY_STORE_SERVE":"true","GORILLA_ALLOW_ANONYMOUS_PUBLIC_PROJECTS":true,"GORILLA_ARTIFACT_GC_ENABLED":"true","GORILLA_DATA_RETENTION_PERIOD":"96h","GORILLA_TRACER":"otlp+grpc://wandb-otel-daemonset:4317?trace_ratio=1","SERVER_FLAG_NAMED_WORKSPACES_AVAILABLE":true,"SUPPORTED_FILE_STORES":"s3://3IAXODZ870OCD6TCFIAD:[email protected]/team-a","TAG_CLOUD":"FLAMA_CLOUD","TAG_CUSTOMER_NS":"FLAMA","TAG_ENV":"HOME_SWEEET_HOME"},"host":"http://wandb.home.lab","license":"***","mysql":{"database":"wandb","host":"mysql.home.lab","password":"***","port":3306,"user":"wandb"},"operator":{"apiVersion":"apps.wandb.com/v1","namespace":"wandb"},"weave-trace":{"enabled":false}},"ingress":{"annotations":{"nginx.ingress.kubernetes.io/proxy-body-size":"64m"},"class":"nginx"},"mysql":{"install":false},"parquet":{"image":{"repository":"wandb/local","tag":"0.63.0"}},"weave":{"image":{"repository":"wandb/local","tag":"0.63.0"},"install":true,"persistence":{"accessMode":"ReadWriteOnce"}},"weave-trace":{"image":{"repository":"wandb/weave-trace","tag":"0.63.0"},"install":false}}}}
2024-12-06T14:06:47Z    INFO    Active spec found       {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "WeightsAndBiases": {"name":"wandb","namespace":"default"}, "namespace": "default", "name": "wandb", "reconcileID": "91025dd8-d0ea-4f04-ae1a-308a7fcdda08", "spec": {"metadata":{"channelId":"f0081406-b0a7-4c51-9c2d-1b911aa68b09","channelInheritsFrom":"b56e1972-3c78-4de0-af90-e3597bb0785a","channelName":"OpenShift - Flamarion","releaseCreatedAt":"2024-12-06T00:52:10.678Z","releaseExtends":"6368f36f-5854-445f-a460-f8ea3fc6d799","releaseId":"e47c97cf-53b9-4965-9246-9e0aa32831df","releaseName":"v202412-5.16"},"chart":{"url":"https://charts.wandb.ai","name":"operator-wandb","version":"0.18.17","password":"","username":""},"values":{"app":{"image":{"repository":"wandb/local","tag":"0.63.0"}},"console":{"image":{"repository":"wandb/console","tag":"2.13.1"}},"flat-run-fields-updater":{"image":{"tag":"0.63.0"}},"global":{"bucket":{"accessKey":"***","name":"pves3.home.lab","path":"wandb","provider":"s3","region":"default","secretKey":"wsnNA1Vq2RHbKdrXSTaw0a09h79QQBZk0AXRUMNY"},"extraEnv":{"ENABLE_REGISTRY_UI":true,"GORILLA_ACTIVITY_STORE_BACKFILL_ENABLE":"true","GORILLA_ACTIVITY_STORE_ENABLE":"true","GORILLA_ACTIVITY_STORE_SERVE":"true","GORILLA_ALLOW_ANONYMOUS_PUBLIC_PROJECTS":true,"GORILLA_ARTIFACT_GC_ENABLED":"true","GORILLA_DATA_RETENTION_PERIOD":"96h","GORILLA_TRACER":"otlp+grpc://wandb-otel-daemonset:4317?trace_ratio=1","SERVER_FLAG_NAMED_WORKSPACES_AVAILABLE":true,"SUPPORTED_FILE_STORES":"s3://3IAXODZ870OCD6TCFIAD:[email protected]/team-a","TAG_CLOUD":"FLAMA_CLOUD","TAG_CUSTOMER_NS":"FLAMA","TAG_ENV":"HOME_SWEEET_HOME"},"host":"http://wandb.home.lab","license":"***","mysql":{"database":"wandb","host":"mysql.home.lab","password":"***","port":3306,"user":"wandb"},"operator":{"apiVersion":"apps.wandb.com/v1","namespace":"wandb"},"weave-trace":{"enabled":false}},"ingress":{"annotations":{"nginx.ingress.kubernetes.io/proxy-body-size":"64m"},"class":"nginx"},"mysql":{"install":false},"parquet":{"image":{"repository":"wandb/local","tag":"0.63.0"}},"weave":{"image":{"repository":"wandb/local","tag":"0.63.0"},"install":true,"persistence":{"accessMode":"ReadWriteOnce"}},"weave-trace":{"image":{"repository":"wandb/weave-trace","tag":"0.63.0"},"install":false}}}}
2024-12-06T14:06:47Z    INFO    No changes found        {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases", "WeightsAndBiases": {"name":"wandb","namespace":"default"}, "namespace": "default", "name": "wandb", "reconcileID": "91025dd8-d0ea-4f04-ae1a-308a7fcdda08"}

Stop the operator after a CTRL+C

^C2024-12-06T14:06:52Z  INFO    Stopping and waiting for non leader election runnables
2024-12-06T14:06:52Z    INFO    Stopping and waiting for leader election runnables
2024-12-06T14:06:52Z    INFO    Shutdown signal received, waiting for all workers to finish     {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases"}
2024-12-06T14:06:52Z    INFO    All workers finished    {"controller": "weightsandbiases", "controllerGroup": "apps.wandb.com", "controllerKind": "WeightsAndBiases"}
2024-12-06T14:06:52Z    INFO    Stopping and waiting for caches
2024-12-06T14:06:52Z    INFO    Stopping and waiting for webhooks
2024-12-06T14:06:52Z    INFO    Stopping and waiting for HTTP servers
2024-12-06T14:06:52Z    INFO    controller-runtime.metrics      Shutting down metrics server with timeout of 1 minute
2024-12-06T14:06:52Z    INFO    shutting down server    {"name": "health probe", "addr": "[::]:8081"}
2024-12-06T14:06:52Z    INFO    Wait completed, proceeding to shutdown the manager