fix: add using include/exclude kinds and namespaces

[afdesk]

Problem

After v0.51.0 to successfully scan a Kubernetes cluster, Trivy must be executed under a role that has read permissions at the cluster scope.

If a user tries to run trivy k8s under a limited account there will be a lot of error message (ex here).

I can reproduce it for limiteduser with the role:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

trivy k8s --report summary --disable-node-collector --kubeconfig ./k8s/configs/limiteduser-kubeconfig --include-namespaces default --include-kinds pod 
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: apps/v1, Resource=deployments - deployments.apps is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=pods - pods is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: apps/v1, Resource=replicasets - replicasets.apps is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=replicationcontrollers - replicationcontrollers is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"replicationcontrollers\" in API group \"\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: apps/v1, Resource=statefulsets - statefulsets.apps is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"statefulsets\" in API group \"apps\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: apps/v1, Resource=daemonsets - daemonsets.apps is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"daemonsets\" in API group \"apps\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: batch/v1, Resource=cronjobs - cronjobs.batch is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"cronjobs\" in API group \"batch\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: batch/v1, Resource=jobs - jobs.batch is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"jobs\" in API group \"batch\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=services - services is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"services\" in API group \"\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=serviceaccounts - serviceaccounts is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"serviceaccounts\" in API group \"\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=configmaps - configmaps is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"configmaps\" in API group \"\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: rbac.authorization.k8s.io/v1, Resource=roles - roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"roles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: rbac.authorization.k8s.io/v1, Resource=rolebindings - rolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"rolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"
2024-08-27T00:25:22+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: networking.k8s.io/v1, Resource=networkpolicies - networkpolicies.networking.k8s.io is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"networkpolicies\" in API group \"networking.k8s.io\" at the cluster scope"
2024-08-27T00:25:23+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: networking.k8s.io/v1, Resource=ingresses - ingresses.networking.k8s.io is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"ingresses\" in API group \"networking.k8s.io\" at the cluster scope"
2024-08-27T00:25:23+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=resourcequotas - resourcequotas is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"resourcequotas\" in API group \"\" at the cluster scope"
2024-08-27T00:25:23+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=limitranges - limitranges is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"limitranges\" in API group \"\" at the cluster scope"
2024-08-27T00:25:23+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: rbac.authorization.k8s.io/v1, Resource=clusterroles - clusterroles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"
2024-08-27T00:25:23+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: rbac.authorization.k8s.io/v1, Resource=clusterrolebindings - clusterrolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"
2024-08-27T00:25:24+06:00	ERROR	Unable to list resources	error="failed listing resources for gvr: /v1, Resource=nodes - nodes is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
2024-08-27T00:25:24+06:00	ERROR	Unable to list node resources	error="nodes is forbidden: User \"system:serviceaccount:default:limiteduser\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"

Reason

Trivy must be able to access information about all cluster resources, including pods, deployments etc.

trivy-kubernetes/pkg/trivyk8s/trivyk8s.go

Lines 174 to 187 in b070991

	for _, gvr := range grvs {
	dclient := c.getDynamicClient(gvr)
	resources, err := dclient.List(ctx, v1.ListOptions{})
	if err != nil {
	lerr := fmt.Errorf("failed listing resources for gvr: %v - %w", gvr, err)
	if errors.IsNotFound(err) || errors.IsForbidden(err) {
	slog.Error("Unable to list resources", "error", lerr)
	continue
	}
	return nil, lerr
	}

Flags include/exclude kinds and namespaces are used only for the result filter.

trivy-kubernetes/pkg/trivyk8s/trivyk8s.go

Lines 189 to 206 in b070991

	for _, resource := range resources.Items {
	if c.ignoreResource(resource) {
	continue
	}
	// if excludeOwned is enabled and the resource is owned by built-in workload, then we skip it
	if c.excludeOwned && c.hasOwner(resource) {
	continue
	}
	// filter resources by kind
	if FilterResources(c.includeKinds, c.excludeKinds, resource.GetKind()) {
	continue
	}
	// filter resources by namespace
	if FilterResources(c.includeNamespaces, c.excludeNamespaces, resource.GetNamespace()) {
	continue
	}

Solution

This PR suggests next solution:

1. Collect scannable resources based on --include-kinds/--exclude-kinds flags.
2. Collect scannable namespaces based on --include-namespaces/--exclude-namespaces flags. If the namespaces are not provided.
3. Run Trivy for resources under each namespace.
4. If resources and namespaces are not provided, Trivy will run as currently at the cluster scope".

note: a part of code for exclude-namespaces isn't implemented yet. I'll add it if the idea is OK for you.

Result:

$ ./trivy k8s --report summary --disable-node-collector --kubeconfig ./k8s/configs/limiteduser-kubeconfig --include-namespaces default --include-kinds pod

1 / 1 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 0 p/s

Summary Report for default


Workload Assessment
┌───────────┬────────────────────────────────────┬───────────────────────────────┬────────────────────┬───────────────────┐
│ Namespace │              Resource              │        Vulnerabilities        │ Misconfigurations  │      Secrets      │
│           │                                    ├─────┬──────┬──────┬──────┬────┼───┬───┬───┬────┬───┼───┬───┬───┬───┬───┤
│           │                                    │  C  │  H   │  M   │  L   │ U  │ C │ H │ M │ L  │ U │ C │ H │ M │ L │ U │
├───────────┼────────────────────────────────────┼─────┼──────┼──────┼──────┼────┼───┼───┼───┼────┼───┼───┼───┼───┼───┼───┤
│ default   │ Pod/my-web-deploy-6dbcdb8c54-kbptl │ 157 │ 1336 │ 3108 │ 1529 │ 85 │   │ 1 │ 4 │ 10 │   │   │   │   │   │   │
└───────────┴────────────────────────────────────┴─────┴──────┴──────┴──────┴────┴───┴───┴───┴────┴───┴───┴───┴───┴───┴───┘
Severities: C=CRITICAL H=HIGH M=MEDIUM L=LOW U=UNKNOWN


Infra Assessment
┌───────────┬──────────┬───────────────────┬───────────────────┬───────────────────┐
│ Namespace │ Resource │  Vulnerabilities  │ Misconfigurations │      Secrets      │
│           │          ├───┬───┬───┬───┬───┼───┬───┬───┬───┬───┼───┬───┬───┬───┬───┤
│           │          │ C │ H │ M │ L │ U │ C │ H │ M │ L │ U │ C │ H │ M │ L │ U │
└───────────┴──────────┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘
Severities: C=CRITICAL H=HIGH M=MEDIUM L=LOW U=UNKNOWN

[simar7]

FYI this might be related aquasecurity/trivy#7107

[afdesk]

FYI this might be related aquasecurity/trivy#7107

thanks. I saw it
this fix just skips the permission error: https://github.com/aquasecurity/trivy-kubernetes/pull/374/files