failed to watch file "/var/log/pods/...*57c7c0d33/docker-build/0.log": no space left on device

[ImperrattoR]

Hello,

we are encountering an error in the OKD environment.
OKD version: 4.14.0-0.okd-2024-01-06-084517

In the logs of pods, we are encountering the error 'failed to watch file "/var/log/pods/...*57c7c0d33/docker-build/0.log": no space left on the device.'

We figured out that the problem is due to the value find /proc/*/fd -user 0 -lname anon_inode:inotify -printf '%hinfo/%f\n' | xargs cat | grep '^inotify' -c' reaching the maximum user watches fs.inotify.max_user_watches` on one of the worker nodes.

On Openshift cluster we double the value with node tuning but these fix only works on Openshift clusters.

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-node-custom
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift node profile with an additional kernel parameter
      [sysctl]
      fs.inotify.max_user_watches=163840
    name: openshift-node-custom
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker"
    priority: 20
    profile: openshift-node-custom
    operand:
      tunedConfig:
        reapply_sysctl: false
      debug: false

So my question is how can I double the value fs.inotify.max_user_watches on our OKD cluster?

Thanks for help.

Best regards, Mike Gavez

[vrutkovs]

On Openshift cluster we double the value with node tuning but these fix only works on Openshift clusters.

That sounds like an NTO bug - it should apply sysctls correctly, regardless of it being OCP or OKD

[ALL-SPACE-Anas]

@ImperrattoR I'm in the same boat. Did you manage to get this issue resolved? If yes, please share the fix.

[ImperrattoR]

Later on the new OKD version, I have applied the tuned file below, and it worked.

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-node-custom
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift node profile with an additional kernel parameter
      [sysctl]
      fs.inotify.max_user_watches=655360
    name: openshift-node-custom
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker"
    priority: 20
    profile: openshift-node-custom
    operand:
      tunedConfig:
        reapply_sysctl: false
      debug: false

[ALL-SPACE-Anas]

it's not working on my 4.15 cluster. I suspect you have it working on OKD4.17?

[ImperrattoR]

Currently, I have an OKD 4.17 cluster, but it definitely worked on 4.15.