Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected EOF log and vulnerability report no data #1792

Open
chary1112004 opened this issue Jan 22, 2024 · 6 comments · May be fixed by #2191
Open

Unexpected EOF log and vulnerability report no data #1792

chary1112004 opened this issue Jan 22, 2024 · 6 comments · May be fixed by #2191
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence. target/kubernetes Issues relating to kubernetes cluster scanning

Comments

@chary1112004
Copy link

chary1112004 commented Jan 22, 2024 •
edited
Loading

What steps did you take and what happened:

Hi,

We are facing issue the vulnerability report no data and log file contains Unexpected EOF

{"level":"error","ts":"2024-01-22T02:24:27Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-59cb6b98d4","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-59cb6b98d4","reconcileID":"054a1f46-d3f3-43a0-a5a5-c8abb278627b","error":"unexpected EOF","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

value file:

operator:
  # scanJobsConcurrentLimit the maximum number of scan jobs create by the operator
  scanJobsConcurrentLimit: 1

  # -- scanJobTimeout the length of time to wait before giving up on a scan job
  scanJobTimeout: 20m
  
  # -- infraAssessmentScannerEnabled the flag to enable infra assessment scanner
  infraAssessmentScannerEnabled: false

  # -- controllerCacheSyncTimeout the duration to wait for controller resources cache sync (default: 5m).
  controllerCacheSyncTimeout: "10m"

serviceMonitor:
  enabled: true
  labels:
    release: prometheus

resources:
  # -- We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  requests:
    cpu: 15m
    memory: 250Mi

trivyOperator:
  # -- scanJobPodTemplateContainerSecurityContext SecurityContext the user wants the scanner and node collector containers (and their
  # initContainers) to be amended with.
  scanJobPodTemplateContainerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    privileged: false
    readOnlyRootFilesystem: true
    # -- For filesystem scanning, Trivy needs to run as the root user
    runAsUser: 0

trivy:
  # -- slow this flag is to use less CPU/memory for scanning though it takes more time than normal scanning. It fits small-footprint
  slow: false

  # -- timeout is the duration to wait for scan completion.
  timeout: "20m0s"

  # -- command. One of `image`, `filesystem` or `rootfs` scanning, depending on the target type required for the scan.
  # For 'filesystem' and `rootfs` scanning, ensure that the `trivyOperator.scanJobPodTemplateContainerSecurityContext` is configured
  # to run as the root user (runAsUser = 0).
  command: filesystem
  
  # -- resources resource requests and limits for scan job containers
  resources:
    requests:
      cpu: 100m
      memory: 1000M
      # ephemeralStorage: "2Gi"
    limits:
      cpu: 500m
      memory: 2000M
      # ephemeralStorage: "2Gi"

Any suggestion to resolve this issue would be very much appreciated!

Thanks!

Environment:

  • Trivy-Operator version (use trivy-operator version): 0.20.1
  • Kubernetes version (use kubectl version): 1.25
  • OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc):
@chary1112004 chary1112004 added the kind/bug Categorizes issue or PR as related to a bug. label Jan 22, 2024
@chen-keinan
Copy link
Contributor

@chary1112004 is the pod still running can you get it output
kubectl logs <pod name> -n trivy-system

@chary1112004
Copy link
Author

chary1112004 commented Jan 23, 2024 •
edited
Loading

@chary1112004 is the pod still running can you get it output kubectl logs <pod name> -n trivy-system

You mean about scan-job pod or trivy-operator pod?
If you mean about trivy-operator pod then yes, it is still running

@chen-keinan
Copy link
Contributor

@chary1112004 is the pod still running can you get it output kubectl logs <pod name> -n trivy-system

You mean about scan-job pod or trivy-operator pod? If you mean about trivy-operator pod then yes, it is still running

I mean for the scan-job

@chary1112004
Copy link
Author

@chary1112004 is the pod still running can you get it output kubectl logs <pod name> -n trivy-system

You mean about scan-job pod or trivy-operator pod? If you mean about trivy-operator pod then yes, it is still running

I mean for the scan-job

yes, scan-job is in status Completed 0/1. However I do not store log of this pod.

@chary1112004 chary1112004 mentioned this issue Feb 1, 2024
Closed
@chen-keinan chen-keinan added target/kubernetes Issues relating to kubernetes cluster scanning priority/backlog Higher priority than priority/awaiting-more-evidence. labels Feb 19, 2024
@daanschipper
Copy link

daanschipper commented May 30, 2024 •
edited
Loading

I'm facing the same issue, possibly related to #591. It seems the pod is terminated before the stdout buffer is flushed, resulting in the unexpected EOF. I tried both setting scanJobCompressLogs to false and true, doesn't seem to make a difference as the output still needs to be flushed before the container is stopped. Running the same command locally does output the full json.

kubectl logs -n trivy-system pod/scan-vulnerabilityreport-85fd8c6947-rfr66 -c <container>

  ... omitted
  {
    "VulnerabilityID": "DLA-3412-1",
    "VendorIDs": [
      "DLA-3412-1"
    ],
    "PkgID": "tzdata@2021a-0+deb10u6",
    "PkgName": "tzdata",
    "PkgIdentifier": {
      "PURL": "pkg:deb/debian/tzdata@2021a-0%2Bdeb10u6?arch=all\u0026distro=debian-%

The other issue suggests adding a sleep to allow the full report to be outputted, but it seems this cannot be configured. Probably a better solution would be to use sync.

@daanschipper daanschipper mentioned this issue Jul 19, 2024
Open
daanschipper added a commit to daanschipper/trivy-operator that referenced this issue Jul 19, 2024
@daanschipper
fix: sync stdout buffer to file
b950fc1 
The trivy command is completed and as it is the main process the entire container is stopped before the stdout
buffer is cleared, resulting in malformed output.

Fixes aquasecurity#1792.
@daanschipper daanschipper linked a pull request Jul 19, 2024 that will close this issue
Open
5 tasks
daanschipper added a commit to daanschipper/trivy-operator that referenced this issue Jul 24, 2024
@daanschipper
fix: sync stdout buffer to file
8e71ce9 
The trivy command is completed and as it is the main process the entire container is stopped before the stdout
buffer is cleared, resulting in malformed output.

Fixes aquasecurity#1792.
daanschipper added a commit to daanschipper/trivy-operator that referenced this issue Jul 24, 2024
@daanschipper
fix: sync stdout buffer to file
f933450 
The trivy command is completed and as it is the main process the entire container is stopped before the stdout
buffer is cleared, resulting in malformed output.

Fixes aquasecurity#1792.
daanschipper added a commit to daanschipper/trivy-operator that referenced this issue Aug 19, 2024
@daanschipper
fix: sync stdout buffer to file
0eddda2 
The trivy command is completed and as it is the main process the entire container is stopped before the stdout
buffer is cleared, resulting in malformed output.

Fixes aquasecurity#1792.
@hubertbarc-rho
Copy link

Having the same issue here although my error is a bit different

{"level":"error","ts":"2024-09-05T02:56:27Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-7955c58bd","namespace":"security-tools"},"namespace":"security-tools","name":"scan-vulnerabilityreport-7955c58bd","reconcileID":"416f2e4e-a244-408c-a2e6-fbdc6fb644b4","error":"unexpected end of JSON input","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

daanschipper added a commit to daanschipper/trivy-operator that referenced this issue Oct 11, 2024
@daanschipper
fix: sync stdout buffer to file
1ed8a3e 
The trivy command is completed and as it is the main process the entire container is stopped before the stdout
buffer is cleared, resulting in malformed output.

Fixes aquasecurity#1792.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence. target/kubernetes Issues relating to kubernetes cluster scanning
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: sync stdout buffer to file daanschipper/trivy-operator
4 participants
@daanschipper @chen-keinan @chary1112004 @hubertbarc-rho