Run kube-bench on individual nodes

[lizrice]

Right now I believe the starboard kube-bench command runs kube-bench on all nodes. Would it be helpful to be able to specify a particular node, for example starboard kube-bench node/mynode?

[hiropinponpan]

I think it is better to be able to specify the node with kube-bench. Because,
When I run starboard kube-bench on an eks cluster that runs both ec2 and fargate node types, the kube-bench job scheduled on the fargate node results in an error.
The kube-bench job scheduled on the ec2 node has been completed normally up to the report output.
below is the log.

$ kubectl get node -o wide
NAME                                                   STATUS   ROLES    AGE     VERSION              INTERNAL-IP      EXTERNAL-IP     OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
fargate-ip-192-168-146-25.us-east-2.compute.internal   Ready    <none>   60m     v1.17.6-eks-4e7f64   192.168.146.25   <none>          Amazon Linux 2   4.14.177-139.254.amzn2.x86_64   containerd://1.3.2
ip-192-168-2-22.us-east-2.compute.internal             Ready    <none>   4d3h    v1.17.9-eks-4c6976   192.168.2.22     52.14.179.145   Amazon Linux 2   4.14.186-146.268.amzn2.x86_64   docker://19.3.6
ip-192-168-32-89.us-east-2.compute.internal            Ready    <none>   5m42s   v1.17.9-eks-4c6976   192.168.32.89    3.21.44.131     Amazon Linux 2   4.14.186-146.268.amzn2.x86_64   docker://19.3.6

$ starboard kube-bench -v 3
I0811 18:30:36.848150   17115 runner.go:79] Running task and waiting forever
I0811 18:30:36.848845   17115 runnable_job.go:47] Creating runnable job: starboard/6e3dc92a-696b-44b0-8463-7cfcbbd195d3
I0811 18:30:36.849224   17115 runner.go:79] Running task and waiting forever
I0811 18:30:36.849241   17115 runnable_job.go:47] Creating runnable job: starboard/5506ca63-430b-4cfc-84ed-98ff01ae89d9
I0811 18:30:36.849267   17115 runner.go:79] Running task and waiting forever
I0811 18:30:36.849278   17115 runnable_job.go:47] Creating runnable job: starboard/e7fc48ec-42f9-4895-8375-8595515a3cac
I0811 18:30:36.880102   17115 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0811 18:30:36.883949   17115 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0811 18:30:36.880990   17115 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0811 18:30:36.884750   17115 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0811 18:30:36.882283   17115 reflector.go:207] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0811 18:30:36.885198   17115 reflector.go:243] Listing and watching *v1.Job from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156
I0811 18:30:36.998669   17115 runnable_job.go:73] Stopping runnable job on task failure with status: Failed
I0811 18:30:36.999629   17115 runner.go:83] Stopping runner on task completion with error: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
E0811 18:30:36.999921   17115 kube_bench.go:61] Error while running kube-bench on node: fargate-ip-192-168-146-25.us-east-2.compute.internal: running kube-bench job: job failed: BackoffLimitExceeded: Job has reached the specified backoff limit
I0811 18:30:39.280486   17115 runnable_job.go:70] Stopping runnable job on task completion with status: Complete
I0811 18:30:39.280511   17115 runner.go:83] Stopping runner on task completion with error: <nil>
I0811 18:30:39.314437   17115 scanner.go:87] Getting logs for kube-bench container in job: starboard/e7fc48ec-42f9-4895-8375-8595515a3cac
I0811 18:30:39.354019   17115 scanner.go:72] Deleting job: starboard/e7fc48ec-42f9-4895-8375-8595515a3cac
I0811 18:30:45.130487   17115 runnable_job.go:70] Stopping runnable job on task completion with status: Complete
I0811 18:30:45.130515   17115 runner.go:83] Stopping runner on task completion with error: <nil>
I0811 18:30:45.159286   17115 scanner.go:87] Getting logs for kube-bench container in job: starboard/6e3dc92a-696b-44b0-8463-7cfcbbd195d3
I0811 18:30:45.200878   17115 scanner.go:72] Deleting job: starboard/6e3dc92a-696b-44b0-8463-7cfcbbd195d3

Below is the error log of a kube-bench job scheduled on the fargate node.

$ kubectl logs 5506ca63-430b-4cfc-84ed-98ff01ae89d9-x8ql9 -n starboard
 Warning  OutOfpods  3m47s  kubelet, fargate-ip-192-168-146-25.us-east-2.compute.internal  Node didn't have enough resource: pods, requested: 1, used: 1, capacity: 1

[danielpacak]

For temporarily tainted nodes or to rescan a particular node we should probably do that. I've created #543 to follow up.