Memory leak with memory mapped file IO

[Lastique]

qBittorrent & operating system versions

qBittorrent: 4.5.5 x86-64, 4.4.1 x86-64
libtorrent-rasterbar: 2.0.9, 2.0.5
Operating System: Kubuntu 22.04
KDE Plasma Version: 5.24.7
KDE Frameworks Version: 5.92.0
Qt Version: 5.15.3
Kernel Version: 6.2.0-1016-lowlatency (64-bit)
Graphics Platform: X11
Processors: 16 × 12th Gen Intel® Core™ i7-12700K
Memory: 31.1 GiB of RAM

What is the problem?

When qBittorrent 4.5.5 is started with "Disk IO type" option in the Advanced tab set to "Default" or "Memory mapped files", qBittorrent starts leaking memory as it is seeding torrents. This also reproduces with qBittorrent 4.4.1, although there was no "Disk IO type" option in that version (I'm assuming, it used memory mapped files internally by default).

In my case, I have 37 torrents, all seeding. A few of the torrents have some files excluded from downloading, but all torrents are at 100% (i.e. everything selected is fully downloaded). The total size of the torrents is about 280 GiB. Upon starting qBittorrent, it starts seeding the torrents at 40-90 Mbps, and its resident memory consumption (as shown in htop as "RES") starts constantly raising. In a few minutes it reaches around 10 GB and continues on.

Setting "Disk IO type" to "POSIX-compliant" fixes the memory leak. With this setting and no other changes, resident memory for qbittorrent process is around 200MB and not climbing further. Torrent seeding bitrate is about the same as with the memory mapped IO.

Steps to reproduce

1. In the Advanced options, set "Disk IO type" to "Default" or "Memory mapped files".
2. Have a few large torrents with a rather high seeding rate.
3. Start qBittorrent and start seeding.
4. Watch qbittorrent process memory consumption in htop.

Additional context

qBittorrent 4.5.5 is installed from this PPA (package version is 1:4.5.5.99~202308312232-7899-7d7097b02~ubuntu22.04.1). It comes with libtorrent-rasterbar 2.0.9.git20230830.3a44a5a78e-1ppa1~22.04. I installed this version in attempt to work around the problem.

qBittorrent 4.4.1 is the version that is available in Kubuntu 22.04 repository (package version is 4.4.1-2). It comes with libtorrent-rasterbar 2.0.5-5. This is the version I originally installed and with which I first had the problem.

In the Advanced options, "Physical memory (RAM) usage limit" is set to 512 MiB (which is the default). This limit is definitely exceeded with memory mapped IO.

Log(s) & preferences file(s)

qBittorrent.conf.gz

[Lastique]

Can you reproduce with qBit v4.6.0

There is no 4.6.0 package in the qbittorrent-stable PPA. I've installed the 4.6.0~202307200732-8070-2c08dc9da~ubuntu22.04.1 package from qbittorrent-unstable PPA. The problem reproduces with it.

with default file pool size 100

If you mean "File pool size" in the Advanced settings, it was set to 256 before. Setting it to 100 does not seem to have any effect on the memory leak.

[Lastique]

Here's my config file with which the problem reproduces.

qBittorrent.conf.gz

[luzpaz]

@Lastique can you use one of the other non-PPA 4.6.0 installs to check this on ?

[Lastique]

Sorry, but no. Due to various issues (this memory leak, low download/upload speeds, .part files, file selection issues), I switched to a different torrent client.

Also, in general I avoid using software from non-deb packages.

[vincejv]

@Lastique
I switched to a different torrent client.
can you post here this alternative client to also help other people encountering this mountain of issues with QBT? I'm also having several issues with QBT recently and piling up, like crashing due to OOM kills due to memory leaks and file name encoding issues and it's taking several mos for QBT/Libtorrent team to fix it, hoping for a quick and easy solution.

Is it still based on libtorrent or is it transmission?

[Lastique]

It's KTorrent. It's not without issues either, but those are not as critical as the ones I listed here.

It uses its own implementation of BitTorrent protocol.

[HanabishiRecca]

@luzpaz this is not a bug. This is how libtorrent 2.x works unfortunately.

There are lots of duplicates of the issues here: https://github.com/qbittorrent/qBittorrent/issues?q=is%3Aissue+memory+ram
I think we may want to create a single meta-issue for that and close others as duplicates, if you don't mind.

The root cause is infamous arvidn/libtorrent#6667

[vincejv]

@luzpaz this is not a bug. This is how libtorrent 2.x works unfortunately.

There are lots of duplicates of the issues here: https://github.com/qbittorrent/qBittorrent/issues?q=is%3Aissue+memory+ram I think we may want to create a single meta-issue for that and close others as duplicates, if you don't mind.

The root cause is infamous arvidn/libtorrent#6667

QBT 5.0 is going to drop anytime soon, perhaps Q1-Q2 2024, I hope libtorrent can catch up🙏🙏 (but as per the lt's head dev, it's going to take some time reworking the code 😵😵)

As we've known in the past QBT's dev are pretty strong in their stand on upgrading dependencies unless there's a mob throwing pitch forks, last stunt they pulled was quietly dropping support for Win 8.1 due to QT, hope it won't be the same scenario again the past year wherein they forced Libtorrent v2.

Should keep the complaints coming so they don't pull off the same move again.
Citing: #17545 - Go back to using libtorrent 1.2.x until qBittorrent 5.x

[HanabishiRecca]

I hope libtorrent can catch up

Yeah. You can kinda track the progress in arvidn/libtorrent#7013

[Lastique]

@luzpaz this is not a bug. This is how libtorrent 2.x works unfortunately.

I know I'm not a qBitTorrent user at this point, but that does not sound like a valid excuse to me. If the memory leak is by design of libtorrent 2.x then that design is broken and needs to be fixed. You cannot just dismiss the problem as "not a bug", as it clearly affects many users and definitely not the expected behavior. Read: it is a bug.

Also I'd like to point out that qBitTorrent, as it is distributed in Kubuntu and PPAs (I'm not sure how "official" they are), only comes with libtorrent 2.x. This is the way people will use qBitTorrent, unless they are willing to go the extra mile of building it themselves or using some other build that uses libtorrent 1.x. And even before that, they must first find out that libtorrent 2.x is the cause of their problems, which isn't obvious at all to a new user.

If your (qBitTorrent maintainers) response is that libtorrent 2.x is broken beyond repair and qBitTorrent should only ever be used with libtorrent 1.x then please make that abundantly clear everywhere - on your website, on this GitHub project front page, etc. Also make sure the distro packages and PPAs are built against libtorrent 1.x. Make a configure-time check that prohibits the use of libtorrent 2.x as a build dependency.

As it stands, as long as qBitTorrent + libtorrent 2.x remains a valid combination, I consider problems coming from it to be valid bugs.

[HanabishiRecca]

If the memory leak is by design

It is not a memory leak. This is how memory-mapped files work both in Linux and Windows. Check the issue I linked or use htop/free/whatever yourself to see how much memory is actually used in your system.

You cannot just dismiss the problem as "not a bug"

It may be a problem, but it is not a bug. A bug is when something works not as intended. But current behavior is totally by design.

So it is better not to complain about things you don't understand and read something like https://www.linuxatemyram.com/

[Lastique]

As I said in the problem description, I was using htop to monitor memory consumption. And that memory consumption is very real as it causes OOM handler to kill processes at some point.

A bug is when something works not as intended. But current behavior is totally by design.

Exhausting system memory uncontrollably certainly isn't the intended behavior of a torrent client. Again, if that's the behavior by design then the design is broken.

So it is better not to complain about things you don't understand

Thanks, I do understand a few things, including about memory accounting in Linux. I also understand it when I see that the system runs out of RAM.

[HanabishiRecca]

As I said in the problem description, I was using htop to monitor memory consumption.

Well, then you have not paid enough attention to the actual used metric.

it causes OOM handler to kill processes at some point.

That's an another question. Yes, this thing happening is a bug. But unfortunately this is a bug of the kernel itself. This is not supposed to happen.

Nor qBittorrent or libtorrent control how memory is allocated or deallocated for memory-mapped files. This is 100% the kernel job. And turns out it does not do it well unfortunately.

So yeah, that's why this idea got a pushback. Again, read the discussion in arvidn/libtorrent#6667

[vincejv]

Yeah. You can kinda track the progress in arvidn/libtorrent#7013

I've read the thread, this PR is mostly arvidn addressing issues with FUSE or Network file system performance, not necessarily the RAM or OOM issues. Has someone actually reported the "kernel bug" if it is indeed "an issue" with the kernel?

I'd stick with libtorrentv1 imo, even after that PR.

[HanabishiRecca]

not necessarily the RAM or OOM issues

They are memory-mapped I/O exclusive problems. Switching form it fixes that problems automatically.

Has someone actually reported the "kernel bug" if it is indeed "an issue" with the kernel?

I don't think so. I personally not proficient enough to dive into the kernel source and find why it misbehaves.

But it is not really worth investigating for us, because:

1. Windows is affected too. So there is a dead end here anyway.
2. Has suboptimal performance with HDDs. As such I/O is entirely controlled by an OS, the client can't pick what to read/write next, can't control the disk cache, so it fully relies on OS's heuristics.
3. Has real performance issues with network drives and behaves even worse than above.

So yeah, we definitely want to switch from it entirely. Which means we effectively can only wait. Or help Arvid to implement that new I/O engine, if you can.

[glassez]

Is there any problem with the kernel? Doesn't it behave the way it's supposed to?
The problem is not that "memory mapped files" feature does not work correctly, but that it is used incorrectly. Apparently, @arvidn just overestimated its capabilities, considering that it was some kind of "magic box" that would do all the work itself. But "memory mapped files" is not some kind of high-level tool. It just does its job the way the operating system is supposed to behave, i.e. it just consumes all available resources.
If @arvidn had implemented memory consumption control (as it was before libtorrent 2.0), "memory mapped files" would not have caused any problems.

[HanabishiRecca]

Is there any problem with the kernel? Doesn't it behave the way it's supposed to?

That's arguable indeed. But I'm fairly sure VFS cache should not cause OOM.

considering that it was some kind of "magic box" that would do all the work itself.

That's exactly how it is advertised though.

It just does its job the way the operating system is supposed to behave, i.e. it just consumes all available resources.

True. But again, it is not supposed to be able to cause OOM. Cached FS pages are simply evicted when the actual working set is required by some app.

[HanabishiRecca]

But reading again about the kernel tunables, there is actually some interesting phrasing.

https://docs.kernel.org/admin-guide/sysctl/vm.html#vfs-cache-pressure

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim.

Which actually drives me to a conclusion that OOM happens when application produces large enough amounts of I/O to exceed the kernel's "fair" reclaim rate. And that's why it is hard to reproduce the problem intentionally.

[Lastique]

Memory mapped files are not VFS cache though. If a process maps a file (or its region), the pages backing that mapping become "locked" in the sense that the kernel can no longer reuse those pages for other purposes. Either those pages need to be in the physical RAM or swapped out. This is unlike the pages that are used by the kernel for regular file IO caching.

If a process continuously maps files without unmapping, this will eventually exhaust both physical RAM and swap and cause OOM. This is entirely expected outcome and is not a bug in the kernel. I'm not sure what else was expected to happen.

[HanabishiRecca]

Not really. Again, if you run free -wh, memory-mapped files go into "cache" column, not into "used". Despite being reported as resident memory of the process, they do not decrease amount of "available" memory in the system.

According to my testing with qBt+LT2.0, it is fine to seed large amounts of data, significantly larger than RAM size and not using swap at all. Its behavior is inconsistent though, OOM may not happen for weeks or even months of uptime, despite physical RAM being 100% full all the time. Opening new apps does not cause OOM either, "cache" memory is simply being reclaimed as intended.

Even when 1 single file is larger than the entire RAM, it does not cause OOM. You can easily test it by forcing such torrent recheck. So there is definitely inconsistency in that. I'm not into the details though.

Btw, some people noticed that reducing File pool size value in advanced options helps.
I think reducing the amount of mapped files at a time simply reduces probability of hitting certain I/O threshold, as unmapping reclaims the memory immediately.

[HanabishiRecca]

Forgot to mention, limiting the working set of the process mitigates the issue. Client reaching artifical limit does not cause the kernel to commit OOM on it.

Windows users seem to be happy with in-built qBittorrent option for that. Unfortunately, it does not work on Linux and you need to use external tools for that.

[vincejv]

Unfortunately, it does not work on Linux

Are you referring to this option in QBT? If so, might as well change this option to "applied if libtorrent >= 2.0 and Windows only"

[glassez]

If a process continuously maps files without unmapping, this will eventually exhaust both physical RAM and swap and cause OOM.

It cannot exhaust the swap, since the swap in this case is the mapped files themselves, and not a regular limited-size swap file.

[glassez]

Windows users seem to be happy with in-built qBittorrent option for that.

Limiting the working set is just a workaround. It cannot fully replace the lack of a managed "cache" in libtorrent 2.0.

Unfortunately, it does not work on Linux.

It was supposed to run on Linux.

@qbittorrent/bug-handlers
Can someone confirm that it doesn't work? Then we would need to remove this option on Linux in the same way as we previously did on macOS.

[HanabishiRecca]

Limiting the working set is just a workaround. It cannot fully replace the lack of a managed "cache" in libtorrent 2.0.

Yes, as I said, its performance is suboptimal.

It was supposed to run on Linux.
Can someone confirm that it doesn't work?

Umm, that's what the source tells

qBittorrent/src/app/application.cpp

Line 1167 in 4ba8eaf

// has no effect on linux but it might be meaningful for other OS

And your own conversations

#16874
#16485 (comment)

And I tested it before, it does not work.

[glassez]

Then we would need to remove this option on Linux in the same way as we previously did on macOS.

@Chocobo1, ping.