Issues Configuring Trivy Operator on an Air-Gapped k3s Cluster

[JosebaAndres]

Hello,

I'm trying to configure the Trivy Operator on a k3s cluster in an air-gapped server (no internet access). Here are the steps I've followed:

1. Uploaded the following OCI resources to a server within the cluster using crane:

   • ghcr.io/aquasecurity/trivy-db:2
   • ghcr.io/aquasecurity/trivy-java-db:1
   • ghcr.io/aquasecurity/trivy-checks:0

2. Configured the Trivy Operator to use these resources as follows:

trivy:
  useBuiltinRegoPolicies: "false"
  useEmbeddedRegoPolicies: "true"
  offlineScan: true
  dbRegistry: "[environment registry server]"
  dbRepository: "aquasecurity/trivy-db:2"
  javaDbRegistry: "[environment registry server]"
  javaDbRepository: "aquasecurity/trivy-java-db:1"
policiesBundle:
  registry: "[environment registry server]"
  repository: "aquasecurity/trivy-checks:0"
  insecure: true

The trivy-db database is downloading correctly, but the scanner returns the following error:

{
    "level": "error",
    "ts": "2024-11-13T07:58:58Z",
    "logger": "reconciler.scan job",
    "msg": "Scan job container",
    "job": "trivy-system/scan-vulnerabilityreport-5b675fd86d",
    "container": "kube-prometheus-stack",
    "status.reason": "Error",
    "status.message": "2024-11-13T07:58:55Z FATAL Fatal error image scan error: scan error: unable to initialize a scanner: unable to initialize an image scanner: unable to find the specified image \"quay.io/prometheus-operator/prometheus-operator:v0.75.1\" in [\"docker\" \"containerd\" \"podman\" \"remote\"]: 4 errors occurred:\n * docker error: unable to inspect the image (quay.io/prometheus-operator/prometheus-operator:v0.75.1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n* containerd error: containerd socket not found: /run/containerd/containerd.sock\n* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n* remote error: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 10.43.0.10:53: server misbehaving; Get \"http://quay.io/v2/\" dial tcp: lookup quay.io on 10.43.0.10:53: server misbehaving\n\n\n",
    "stacktrace": "github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n /home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353 github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n /home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80 sigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"
}

I thought I needed to configure the socket path to work with the k3s path as documented in the trivy documentation

https://trivy.dev/v0.57/docs/references/troubleshooting/#containerd

But i started getting this new error:

{
  "level": "error",
  "ts": "2024-11-18T13:48:43Z",
  "logger": "reconciler.scan job",
  "msg": "Scan job container",
  "job": "trivy-system/scan-vulnerabilityreport-9d6795d84",
  "container": "proxy",
  "status.reason": "Error",
  "status.message": "2024-11-18T13:48:28Z\tFATAL\tFatal error\timage scan er ││ ror: scan error: unable to initialize a scanner: unable to initialize an image scanner: unable to find the specified image \"kong:3.6\" in [\"docker\" \"containerd\" \"podman\" \"remote\"]: 4 errors occurred:\n\t* docker error: failed to initialize a docker client: unable to  ││ parse docker host `/run/k3s/containerd/containerd.sock`\n\t* containerd error: containerd socket not found: /run/k3s/containerd/containerd.sock\n\t* podman error: unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n ││ t* remote error: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 10.43.0.10:53: server misbehaving; Get \"http://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 10.43.0.10:53: server misbehaving\n\n\n",
  "stacktrace": "github.com/aquasecurity/ ││ trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJob ││ Controller).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/contro ││ [email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/ ││ internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/ ││ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"
}

Does anyone know how I can configure it? Am I approaching this incorrectly?

Thanks in advance!

[JosebaAndres]

I think I’ve found the problem. In the k3s cluster, we have a registry mirror, and I’ve discovered that this is not yet supported in Trivy.

Thank you anyway, and best regards.