Finit leaking inotify watchers?

[milkylainen]

Hello!

I have this situation where triggering finit results in inotify watchers running out.
Various situations can trigger this forever escalating usage, but simplest is probably switching runlevels back and forth.
I'm not 100% sure this is finit causing this and not the local kernel (5.17.x), but I can't find anything regarding this issue in the kernel either.

I've patched the local kernel to just dump the paths of the added watchers (ymmv, but you get the point).
inotify_user.txt

When running a couple of runlevel switches I get this in my kernel logs.

[ 242.679197] Decreasing inotify counts
[ 244.189344] Addding watch for pathname /sys/fs/cgroup/system/gen-ssh-keys/cgroup.events: idx: 65
[ 244.191387] Addding watch for pathname /sys/fs/cgroup/system/mnt-startd/cgroup.events: idx: 66
[ 244.194070] Addding watch for pathname /sys/fs/cgroup/system/mtd-chksum/cgroup.events: idx: 67
[ 244.199392] Addding watch for pathname /sys/fs/cgroup/system/selftest/cgroup.events: idx: 68
[ 244.219467] Addding watch for pathname /sys/fs/cgroup/system/chronyd/cgroup.events: idx: 69
[ 244.222096] Addding watch for pathname /sys/fs/cgroup/system/dnsmasq:dns/cgroup.events: idx: 70
[ 244.224973] Addding watch for pathname /sys/fs/cgroup/system/lldpd/cgroup.events: idx: 71
[ 244.225767] New inotify group idx 11
[ 244.225853] Addding watch for pathname /var/run/netd: idx: 72
[ 244.228959] Addding watch for pathname /sys/fs/cgroup/system/snmpd/cgroup.events: idx: 73
[ 244.231738] Addding watch for pathname /sys/fs/cgroup/system/querierd/cgroup.events: idx: 74
[ 244.235331] Addding watch for pathname /sys/fs/cgroup/system/ssdpd/cgroup.events: idx: 75
[ 301.153872] Decreasing inotify counts
[ 305.303141] Addding watch for pathname /sys/fs/cgroup/system/mnt-startd/cgroup.events: idx: 75
[ 305.307354] Addding watch for pathname /sys/fs/cgroup/system/mtd-chksum/cgroup.events: idx: 76
[ 305.313518] Addding watch for pathname /sys/fs/cgroup/system/selftest/cgroup.events: idx: 77
[ 305.333666] Addding watch for pathname /sys/fs/cgroup/system/chronyd/cgroup.events: idx: 78
[ 305.336029] Addding watch for pathname /sys/fs/cgroup/system/dnsmasq:dns/cgroup.events: idx: 79
[ 305.338938] Addding watch for pathname /sys/fs/cgroup/system/lldpd/cgroup.events: idx: 80
[ 305.339739] New inotify group idx 11
[ 305.339819] Addding watch for pathname /var/run/netd: idx: 81
[ 305.341932] Addding watch for pathname /sys/fs/cgroup/system/snmpd/cgroup.events: idx: 82
[ 305.344739] Addding watch for pathname /sys/fs/cgroup/system/querierd/cgroup.events: idx: 83
[ 305.353630] Addding watch for pathname /sys/fs/cgroup/system/ssdpd/cgroup.events: idx: 84
[ 337.964640] Decreasing inotify counts
[ 341.165331] Addding watch for pathname /sys/fs/cgroup/system/gen-ssh-keys/cgroup.events: idx: 84
[ 341.167410] Addding watch for pathname /sys/fs/cgroup/system/mnt-startd/cgroup.events: idx: 85
[ 341.170227] Addding watch for pathname /sys/fs/cgroup/system/mtd-chksum/cgroup.events: idx: 86
[ 341.174801] Addding watch for pathname /sys/fs/cgroup/system/selftest/cgroup.events: idx: 87
[ 341.197329] Addding watch for pathname /sys/fs/cgroup/system/chronyd/cgroup.events: idx: 88
[ 341.199771] Addding watch for pathname /sys/fs/cgroup/system/dnsmasq:dns/cgroup.events: idx: 89
[ 341.202695] Addding watch for pathname /sys/fs/cgroup/system/lldpd/cgroup.events: idx: 90
[ 341.203580] New inotify group idx 11
[ 341.203659] Addding watch for pathname /var/run/netd: idx: 91
[ 341.206496] Addding watch for pathname /sys/fs/cgroup/system/snmpd/cgroup.events: idx: 92
[ 341.209502] Addding watch for pathname /sys/fs/cgroup/system/querierd/cgroup.events: idx: 93
[ 341.212853] Addding watch for pathname /sys/fs/cgroup/system/ssdpd/cgroup.events: idx: 94

This keeps on increasing until I hit the watcher limit.
Granted, on a normal system this is a pretty big number. So it's not super critical.
For testing I usually lower the watcher limit through procfs:

root@lynx-4d-76-a0:~ # cat /proc/sys/fs/inotify/max_user_instances
128
root@lynx-4d-76-a0:~ # cat /proc/sys/fs/inotify/max_user_watches
8192

Looking at finit, from service.c, service_fork -> cgroup_service -> cgroup_leaf_init which places them all in system as a fallback? And "system" is protected? Protected (active) cgroups afaiu, don't get removed.

Regarding the ever increasing index. I think this has something to do being placed in a new group or similar?
Because I can see resonable filtering added to the iwatch_add function. (Although there are some users of the raw iwatch_add1 function.)

When I hit the roof, I get an ENOSPC from the kernel (inotify_user.c).