Multiple "Out of memory" killings on speedy rsync backup

[brauliobo]

System information

Type	Version/Name
Distribution Name	CachyOS
Distribution Version	rolling latest
Kernel Version	6.18.0
Architecture	amd64
OpenZFS Version	2.3.5

Describe the problem you're observing

In a system with 192gb of RAM and only 25gb used (remaining was cache) during an rsync with speed over 200mb/s (megabytes per second), ZFS was triggering "Out of memory" killings as could be confirmed with dmesg logs. It started with Chrome, Cursor and bigger process but would not stop and kept killing many processes.

Setting primarycache=none or primarycache=metadata workarounds the issue
Users suggested configuring zfs_max_arc, which I didn't try.

The issue is that ZFS should not use that much memory and not checking kernel memory constraints. Also it should consider releasing memory for old stuff, like the Linux block cache.

Ideally it would be releasable memory like with the Linux block cache, and never capable of killing processes.

I can easily imagine ZFS killing a database like PostgreSQL that it is meant to serve. So the defaults zfs_max_arc=0 and all_system_memory - 1GB are really dangerous.

Related discussion: https://www.reddit.com/r/zfs/comments/1pkve4m/out_of_memory_on_fast_backup_operation/

[SinisterCrayon]

Interesting point that you're running CachyOS. Are you using the BORE scheduler? I'm wondering if the BORE scheduler (which prioritizes interactive tasks rather than background) is one of the factors at play here. It would make sense that it would probably prioritize responsiveness of the UI over ARC deflation in high load situations leading to your OOM problems.

Can you test it with a different scheduler for example CFQ or MQ-DEADLINE to see if it resolves your issues? It might cause some occasional UI lagginess (not sure how changing the scheduler will affect the other package optimizations in CachyOS) I'd be interested to see if it does.

EDIT: Looks like CachyOS includes the ADIOS scheduler for heavy I/O loads. I wonder if that would help.