Does Starboard can run with container engine cri-o?

[chansgt]

Hi i got an error when try to run starboard on OKD 3.11 with container engine cri-o. The errors is in below:

Does Starboard can run with container engine cri-o?

[danielpacak]

@chansgt I think this is not related to Kubernetes distribution nor container runtime. It seems that you're running a Kubernetes workload that refers to container image (quay.io/openshift/origin-haproxy-router:v3.11) with old manifest metadata schema (application/vnd.docker.distribution.manifest.v1+prettyjws).

Trivy, which is the default vulnerability scanner used by Starboard, only supports manifest schema v2. We're not going to support old manifest formats in Trivy. /cc @knqyf263

I can reproduce the error by running Trivy directly:

trivy image quay.io/openshift/origin-haproxy-router:v3.11
2021-01-04T16:32:59.131+0100	INFO	Need to update DB
2021-01-04T16:32:59.132+0100	INFO	Downloading DB...
19.58 MiB / 19.58 MiB [-------------------------------------------------------------------------------------------------] 100.00% 21.25 KiB p/s 15m44s
2021-01-04T16:48:46.190+0100	FATAL	unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
	* unable to inspect the image (quay.io/openshift/origin-haproxy-router:v3.11): Error: No such image: quay.io/openshift/origin-haproxy-router:v3.11
	* unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

[chansgt]

okk thanks for your info @danielpacak .

I try to using starboard cli to scan the deployment but the result is error like the picture below

Do you have any solution for this?

[danielpacak]

Let's keep those discussion focused on one topic at a time. Please file an issue and provide more details about your environment, in particular Kubernetes version / distribution, Starboard release or commit id, as well as any changed to the default config.

There are two flags that help troubleshooting such cases that you can try out:

starboard scan vulnerabilityreports deployment/nginx -n test -v 3 --delete-scan-job=false

This command is more verbose and does not delete the scan job and the controlled pod so you can check their status / logs and see what is going on.

[chansgt]

okk. Thank you for your information. I will try the command.

[chansgt]

Hi @danielpacak ,

I think there is a problem for Starboard CLI version 0.7.1. When i try to scan vulnerability from deployment inside cluster it fails and the error is same like the picture above that i sent to you before. But when i change to version 0.7.0 the scanning is work.

Here is my environment spec:

1. Cluster: OKD 3.11
2. Container Engine: cri-o
3. OS: CentOS Linux release 7.9.2009 (Core)
4. Starboard 0.7.1: {Version:0.7.1 Commit:3c4342d2f344dd3aeea3fb1083c9d7a499bbcf2f Date:2020-12-05T18:05:28Z}
5. Starboard 0.7.0: {Version:0.7.0 Commit:4ab81568c52097bdd16c6b9cb424f5338a78ba24 Date:2020-12-02T22:52:19Z}

Thank you.

[danielpacak]

I cannot reproduce it with Starboard CLI 0.7.1 on upstream Kubernetes in my environment. @chansgt Would you be able to provide the description of the pod that's create by the scan job? You can find the name of the pod after enabling the verbose (-v 3) output. It's 28eb67be-641d-44d0-9f36-8f8453b72a9e-pgxl9 in the example below. In particular, I'm interested in events associated with that pod to see if/why there's a problem with the scanner. Also, please confirm which container image is run by the nginx deployment and whether it has Running state in your cluster?

$ kubectl starboard init -v 3
I0107 12:03:28.756838   20183 cert_rotation.go:137] Starting client certificate rotation controller
I0107 12:03:28.775932   20183 cr_manager.go:321] Creating CRD "vulnerabilityreports.aquasecurity.github.io"
I0107 12:03:28.786420   20183 cr_manager.go:321] Creating CRD "ciskubebenchreports.aquasecurity.github.io"
I0107 12:03:28.791001   20183 cr_manager.go:321] Creating CRD "kubehunterreports.aquasecurity.github.io"
I0107 12:03:28.796832   20183 cr_manager.go:321] Creating CRD "configauditreports.aquasecurity.github.io"
I0107 12:03:28.802896   20183 cr_manager.go:253] Creating Namespace "starboard"
I0107 12:03:28.816970   20183 cr_manager.go:268] Creating ServiceAccount "starboard/starboard"
I0107 12:03:28.820707   20183 cr_manager.go:285] Creating ClusterRole "starboard"
I0107 12:03:28.824955   20183 cr_manager.go:303] Creating ClusterRoleBinding "starboard"

$ kubectl create deployment nginx --image nginx:1.16

$ kubectl starboard scan vulnerabilityreports deploy/nginx -v 3 --delete-scan-job=false
I0107 12:03:46.727698   20211 cert_rotation.go:137] Starting client certificate rotation controller
I0107 12:03:46.786792   20211 scanner.go:62] Getting Pod template for workload: {Deployment nginx default}
I0107 12:03:46.791773   20211 scanner.go:68] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:false}
I0107 12:03:46.793468   20211 runner.go:79] Running task and waiting forever
I0107 12:03:46.793989   20211 runnable_job.go:77] Creating job: starboard/28eb67be-641d-44d0-9f36-8f8453b72a9e
I0107 12:03:46.807358   20211 reflector.go:175] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125
I0107 12:03:46.807375   20211 reflector.go:211] Listing and watching *v1.Job from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125
I0107 12:04:09.071329   20211 runnable_job.go:112] Stopping runnable job on task completion with status: Complete
I0107 12:04:09.071362   20211 runner.go:83] Stopping runner on task completion with error: <nil>
I0107 12:04:09.071367   20211 scanner.go:98] Scan job completed: starboard/28eb67be-641d-44d0-9f36-8f8453b72a9e
I0107 12:04:09.076978   20211 scanner.go:176] Getting logs for nginx container in job: starboard/28eb67be-641d-44d0-9f36-8f8453b72a9e
I0107 12:04:09.782353   20211 scanner.go:88] Skipping scan job deletion: starboard/28eb67be-641d-44d0-9f36-8f8453b72a9e
I0107 12:04:09.810739   20211 writer.go:71] Creating VulnerabilityReport "default/deployment-nginx-nginx"

$ kubectl describe pod 28eb67be-641d-44d0-9f36-8f8453b72a9e-pgxl9 -n starboard