error while starting otel collector daemon set: open /sys/fs/cgroup/memory/memory.limit_in_bytes: permission denied"

[kiranmenon]

Component(s)

No response

What happened?

Describe the bug
The otel collector pod is failing to start with error compliaining about permisson "open /sys/fs/cgroup/memory/memory.limit_in_bytes: permission denied". I had installed the otel collector via helm chart version 0.139.0. I have other k8s cluster running fine with same helm chart version, also on the same cluster 2/3 daemonset has started and running fine. What could be the permission we are missing on that 1 node? how to handle that via helm?

Steps to reproduce
Hard to reproduce as we are observing it only on 1 of our k8s node so far
Installation step followed:

• helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
• helm -n otel upgrade -i app-metrics open-telemetry/opentelemetry-collector --values otel/otel-collector-daemonset-values.yml
• helm value have mode: daemonset set

What did you expect to see?
otel collector daemonset running on all nodes

What did you see instead?
one daemonset replica failing to init

Collector version

0.139.0

Environment information

Environment

OKE cluster
image pulled from: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.139.0

OpenTelemetry Collector configuration

config:
  receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: app-metrics
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                action: keep
                regex: "true"
              - source_labels: [__meta_kubernetes_pod_ip]
                target_label: __address__
                replacement: $1:9252

  processors:
    k8sattributes: {}
    batch: {}
    resource:
      attributes:
        - action: upsert
          key: service.name
          value: app-metrics

  exporters:
    otlphttp:
      
      endpoint: "<url-to-oci-apm>"
      headers:  
        authorization: "xxx"   
      
  service:
    pipelines:
      metrics:
        receivers: [prometheus]
        processors: [k8sattributes, resource, batch]
        exporters: [otlphttp]

Log output

2025-11-20T05:20:43.107Z    error    service@v0.139.0/service.go:143    error found during service initialization    {"resource": {"service.instance.id": "8741cac2-cdd4-411f-b0c0-3faf21b97a36", "service.name": "otelcol-contrib", "service.version": "0.139.0"}, "error": "failed to build pipelines: failed to create \"memory_limiter\" processor, in pipeline \"traces\": failed to get total memory, use fixed memory settings (limit_mib): open /sys/fs/cgroup/memory/memory.limit_in_bytes: permission denied"}
go.opentelemetry.io/collector/service.New.func1
    go.opentelemetry.io/collector/service@v0.139.0/service.go:143
go.opentelemetry.io/collector/service.New
    go.opentelemetry.io/collector/service@v0.139.0/service.go:213
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
    go.opentelemetry.io/collector/otelcol@v0.139.0/collector.go:203
go.opentelemetry.io/collector/otelcol.(*Collector).Run
    go.opentelemetry.io/collector/otelcol@v0.139.0/collector.go:324
go.opentelemetry.io/collector/otelcol.NewCommand.func1
    go.opentelemetry.io/collector/otelcol@v0.139.0/command.go:39
github.com/spf13/cobra.(*Command).execute
    github.com/spf13/cobra@v1.10.1/command.go:1015
github.com/spf13/cobra.(*Command).ExecuteC
    github.com/spf13/cobra@v1.10.1/command.go:1148
github.com/spf13/cobra.(*Command).Execute
    github.com/spf13/cobra@v1.10.1/command.go:1071
main.runInteractive
    github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:70
main.run
    github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:10
main.main
    github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:63
runtime.main
    runtime/proc.go:285
Error: failed to build pipelines: failed to create "memory_limiter" processor, in pipeline "traces": failed to get total memory, use fixed memory settings (limit_mib): open /sys/fs/cgroup/memory/memory.limit_in_bytes: permission denied
2025/11/20 05:20:43 collector server run finished with error: failed to build pipelines: failed to create "memory_limiter" processor, in pipeline "traces": failed to get total memory, use fixed memory settings (limit_mib): open /sys/fs/cgroup/memory/memory.limit_in_bytes: permission denied
stream closed: EOF for otel/gitlab-runner-metrics-opentelemetry-collector-agent-m9v6z (opentelemetry-collector)

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}