Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in v5.0.0 #21502

Closed
kieraneglin opened this issue Oct 4, 2024 · 73 comments
Closed

Memory leak in v5.0.0 #21502

kieraneglin opened this issue Oct 4, 2024 · 73 comments

Comments

@kieraneglin
Copy link

qBittorrent & operating system versions

qBittorrent: v5.0.0 x64
OS: It's running in an Arch Linux-based Docker container. The host is Unraid 6.12.13
Qt: 6.7.3
Libtorrent: 2.0.10.0
Boost: 1.86.0
OpenSSL: 3.3.2
zlib: 1.3.1

What is the problem?

Based on memory usage, there appears to be a slow leak introduced on my system by the 5.0.0 upgrade. I was running 4.6.7 beforehand (I think) without any issue.

The memory usage constantly increases and appears to be independent from torrent activity overall. There are spikes in memory when a torrent is actively downloading, but the overall trend slowly marches upwards as shown in the usage screenshot below. Restarting instantly fixes the issue in the immediate, but the usage will then start climbing from that point.

I understand that memory leaks SUCK to debug so I'm happy to run any diagnostic/troubleshooting steps you'd like!

Steps to reproduce

  1. Start qBittorrent
  2. Wait and see memory increasing

Additional context

Screenshot 2024-10-04 at 8 23 56 AM

Log(s) & preferences file(s)

watched_folders.json is empty (contents is {})
qBittorrent.conf.txt

@HanabishiRecca

This comment was marked as outdated.

@kieraneglin
Copy link
Author

kieraneglin commented Oct 4, 2024

Sure, but what would the explanation be for this only emerging with the upgrade from 4.6.7 to 5.0.0? Is it purely a reporting issue?

My logging history doesn't go back far enough for me to prove it, but I check my metrics dashboard a few times per week and this issue can be directly correlated with my upgrade to v5. That's not to say that you're wrong! But from where I'm standing this is a novel behaviour.

@TheBeastBHR
Copy link

I had the same experience, its using 22.5GB RAM very quickly before I ended the task from task manager (it wouldn't even exit normally and won't open a window to the app).
I used 4.6.6-4.6.7 extensively and didn't have this issue before.

@SanderBouwhuis
Copy link

This problem has been known for years now. Does v5 force the use of Libtorrent v2, or can I upgrade to v5 safely using Libtorrent v1.2?
I need to switch bittorrent clients every 5 to 10 years because projects somehow seem to screw up. This trend started with uTorrent.

@ArcticGems
Copy link

ArcticGems commented Oct 5, 2024

@SanderBouwhuis
there are 2 variation of each release, for example:
5.0.0
5.0.0 (qt6 lt20)

First one uses libtorrent 1.2 (both uses qt6)

In the next major release of qB (5.1 not 5.0.x), will have a workaround for the lt 2.0 issue.

@Dekumori
Copy link

Dekumori commented Oct 5, 2024

Can confirm. I was on 4.6.7 up until 10/01 and you can very clearly see where the upgrade took place. My Zabbix worker ended up crashing due to the memory usage spike and the memory usage has been erratic ever since.

image

@TGKx
Copy link

TGKx commented Oct 5, 2024

Can confirm same 5.0 docker container issue, RES size increased to 2GB of ram from initial usage of ~20MB for a single inactive torrent running over the period of 5-6 days. Going to stay on 4.6.x branch until this is properly fixed.

@ArcticGems
Copy link

ArcticGems commented Oct 5, 2024

@Dekumori @TGKx
What Libtorrent version is mentioned in your qB 5.0 installation: Help->About->Software Used

If Libtorrent 2.0.X, try qB 5.0 release variation with 1.2.X

@TGKx
Copy link

TGKx commented Oct 5, 2024

Qt: | 6.6.3
Libtorrent: | 1.2.19.0
Boost: | 1.84.0
OpenSSL: | 3.3.2
zlib: | 1.3.1

This is from docker image: https://github.com/qbittorrent/docker-qbittorrent-nox/pkgs/container/docker-qbittorrent-nox/281473771?tag=5.0.0-1

@Dekumori
Copy link

Dekumori commented Oct 5, 2024

@ArcticGems I am on Lib 2.0.X. I'll switch releases. I did mitigate the issue by limiting memory usage in my docker compose yml file. My system is stable, though the container is still unstable. Below is the memory usage since the change. I'll update after a few hours running the alternate release.

image

@maxim-mityutko
Copy link

The problem is definitely there. 4.6.7 was working without an issue. 5.0.0 gets OOM killed on my cluster every 8-9 hours, like a clockwork since the update
image

@ArcticGems
Copy link

The problem is definitely there. 4.6.7 was working without an issue. 5.0.0 gets OOM killed on my cluster every 8-9 hours, like a clockwork since the update

@maxim-mityutko
What Libtorrent version is mentioned in your qB 5.0 installation: Help->About->Software Used

If Libtorrent 2.0.X, try qB 5.0 release variation with 1.2.X

@Dekumori
Copy link

Dekumori commented Oct 6, 2024

Confirmed still occurring after switching to Libtorrent 1.2.19.0 release. It's not maxing out because I put a limit on the docker container.

image

@HanabishiRecca
Copy link
Contributor

Confirmed still occurring after switching to Libtorrent 1.2.19.0 release.

Finally a useful comment. There is another known issue:

Does it resembles your use pattern?

@Dekumori
Copy link

Dekumori commented Oct 6, 2024

It looks like both come down to issues with the webui being open. I closed all webui tabs and will monitor for a few hours. If the issue persists, I will restart the container and monitor again. If the issue still persists, I'll restart the server and monitor again. I'll return back with results once the issue is fixed or all of my above steps have been exhausted. I'm happy to try any alternative steps that are suggested as well.

@maxim-mityutko
Copy link

@ArcticGems


qBittorrent was built with the following libraries:

Qt:	6.7.3
Libtorrent:	1.2.19.0
Boost:	1.86.0
OpenSSL:	3.3.2
zlib:	1.3.1.zlib-ng

@maxim-mityutko
Copy link

It looks like both come down to issues with the webui being open. I closed all webui tabs and will monitor for a few hours. If the issue persists, I will restart the container and monitor again. If the issue still persists, I'll restart the server and monitor again. I'll return back with results once the issue is fixed or all of my above steps have been exhausted. I'm happy to try any alternative steps that are suggested as well.

  • The machine from which I usually manage the torrents is turned off for the most part of the day.
  • My instance is running in the pod in Kubernetes, pretty much isolated from anything else
  • I'm using VueTorrent WebUI instead of the native one

@IRainman
Copy link

IRainman commented Oct 7, 2024

Same behaviour in Windows 10 too.
Problem started after I'm updating from version installed from file “qbittorrent_4.6.7_lt20_qt6_x64_setup.exe” to the version “qbittorrent_5.0.0_lt20_qt6_x64_setup.exe”
image

@HanabishiRecca
Copy link
Contributor

Please post full technical details about your system and qBittorent build, as requested by the bug report template.
"Me too" comments without details are not helpful.

@HanabishiRecca
Copy link
Contributor

Thank you.

Problem started after I'm updating from version installed from file “qbittorrent_4.6.7_lt20_qt6_x64_setup.exe” to the version “qbittorrent_5.0.0_lt20_qt6_x64_setup.exe”

Try the regular one instead and see if the problem persists.
https://www.fosshub.com/qBittorrent.html?dwl=qbittorrent_5.0.0_x64_setup.exe

@IRainman
Copy link

IRainman commented Oct 7, 2024

Thank you.

Problem started after I'm updating from version installed from file “qbittorrent_4.6.7_lt20_qt6_x64_setup.exe” to the version “qbittorrent_5.0.0_lt20_qt6_x64_setup.exe”

Try the regular one instead and see if the problem persists. https://www.fosshub.com/qBittorrent.html?dwl=qbittorrent_5.0.0_x64_setup.exe

Thank you, but I don't have any reason to use it. I especially use the version with qt6, lib torrent 2.0, enabled memory mapped files, and SQLite for storing data. Currently, I am downgrading to version 4.6.7 and absolutely all goings fine.
image

@kieraneglin
Copy link
Author

Reverted to 4.6.7 with Libtorrent 2 and the issue does NOT persist. Usage in the graph correlates perfectly with new torrents being downloaded.

Qt: 6.7.2
Libtorrent: 2.0.10.0
Boost: 1.86.0
OpenSSL: 3.3.2
zlib: 1.3.1

Screenshot 2024-10-07 at 9 33 22 AM

@starobots
Copy link

I use the version 1.2 libtorrent of 5.0 qbt, but it cost about 20gb physics memory and all the virtual memory about 30 gb,50 gb in total.
And I installed 4.6.6 back, and it works much better than 5.0.0ver.

@HanabishiRecca
Copy link
Contributor

HanabishiRecca commented Oct 7, 2024

Another possible test: try to reset the client to default settings.
I.e. delete existing client config (make a backup if you want). The config location per platform is described in the wiki.

Also we need:

  1. Reproduction steps. Exactly what you do with the client, maybe some patterns you've noticed.
  2. Logs. Any info as much as possible.
  3. Crash reports, dumps and backtraces, if any.

P.S. If everyone going to flip the table and roll back to older versions, the problem would never be fixed.

@kieraneglin
Copy link
Author

P.S. If everyone going to flip the table and roll back to older versions, the problem would never be fixed.

I'm not sure if this is your intent, but many of your replies are coming off as antagonistic. I've rolled back as a troubleshooting step to gather more details and not because I'm throwing in the towel. As said in my original post I'm more than happy to put in some work to help identify the root cause.

In any case, I'll spin up a new v5.0.0 Docker container which will provide me with a completely blank slate and let you know what I see!

@HanabishiRecca
Copy link
Contributor

HanabishiRecca commented Oct 7, 2024

I've rolled back as a troubleshooting step to gather more details and not because I'm throwing in the towel. As said in my original post I'm more than happy to put in some work to help identify the root cause.

Glad to hear. Due to the comment section being kinda spammy, it's easier to lose track.
That's because 3 comments in a row above my message looked like a throw:

I am downgrading to version 4.6.7
Reverted to 4.6.7
I installed 4.6.6 back


I'll spin up a new v5.0.0 Docker container which will provide me with a completely blank slate and let you know what I see!

Sure. It's a crucial step now to pinpoint the issue and make it reproducible.


And yes, my initial assumption (#21502 (comment)) seems to be wrong now. We just so used to such reports at this point, that anything about "memory leak" triggers that association by reflex.

@kieraneglin
Copy link
Author

Update: A completely fresh qbit 5.0.0 install with no torrents has been slowly increasing in memory usage. It's important to note the scale here - this is showing 62 MB to ~85MB over the course of 2 hours. Far, far slower than the prior ~500 MB per hour I was seeing before. These numbers may be too small as to be useful but I figured I'd update.

I'll keep it running and see what I find

Screenshot 2024-10-07 at 3 27 46 PM

@glassez
Copy link
Member

glassez commented Oct 8, 2024

@stalkerok
I know you maintain big amount of torrents. Have you already switched to qBittorrent v5.0?

@vincejv
Copy link

vincejv commented Oct 8, 2024

hello all, i'm here to confirm this bug, i'm fully aware of the mmap issue w/ LT20, actually i'm one of the people who pleaded to have the LT1.2 option with QBT 5.0 so I'm still using QBT 5.0 with LT1.2 ...

if anyone can recommend a tool to memory profile the client while it's running so I can provide more details? one that works on docker containers/qbitorrent nox.. it's so severe that the machine actually uses up the entire swap memory, this happened on two separate linux machines, one hasn't crashed yet thankfully, but will downgrade to 4.6.7 with lt1.2 for the meantime...

Machine 1:
image

Machine 2:
image

@vincejv
Copy link

vincejv commented Nov 2, 2024

This seems to be highly correlated with the Web UI/API, so I had one instance of QBT running thru Web UI the entire day and the RAM Usage grew until OOM... while the other instance I didn't access the Web UI, just monitored through qbittorrent exporter tool (a Prometheus exporter that polls on QBT API)

The previous memory leak issues that were raised were the browser leaking, but this time the server/the qbittorrent client itself is leaking RAM on the server, I hope you can figure out what's causing this as pretty much this RCE Bug https://news.ycombinator.com/item?id=42004219 caused QBT < 5.0.1 to be outright banned in several popular trackers forcing the usage of QBittorrent 5.0

I would guess @stalkerok isn't affected since he's not using the Web UI.

@vincejv
Copy link

vincejv commented Nov 2, 2024

So to repro this memory leak, here's my checklist

  • Many torrents loaded in the client >400
  • Many trackers on those torrents, mine is around an estimated 200 tracker URLs overall
  • Have an active Web UI open all day long (refresh rate 2000ms)

@stalkerok
Copy link
Contributor

I hope you can figure out what's causing this as pretty much this RCE Bug https://news.ycombinator.com/item?id=42004219 caused QBT < 5.0.1 to be outright banned in several popular trackers

lol 😂
People don't seem to realize that this is not a critical vulnerability at all. I wouldn't call it a vulnerability at all. Somebody's just trying to make a big deal out of it.

I would guess @stalkerok isn't affected since he's not using the Web UI.

Yeah, I don't use the Web UI.

@HanabishiRecca
Copy link
Contributor

The previous memory leak issues that were raised were the browser leaking, but this time the server/the qbittorrent client itself is leaking RAM on the server

The issue mentioned in #21502 (comment) is about memory leaking on the server.

@claabs
Copy link

claabs commented Nov 13, 2024

FYI, I've been on the 5.0.1 linuxserver image for a week now and have not experienced any memory leak issues since upgrading. For me personally, this is resolved.

@osh123
Copy link

osh123 commented Nov 13, 2024

Same for me, since upgrading to 5.0.1 - memory usage is stable.
Using Home Assistant addon image based on linuxserver image.

@vincejv
Copy link

vincejv commented Nov 13, 2024

same, on 5.0.1 LT1.2, it seems like this has been fixed by the following PR

@luzpaz
Copy link
Contributor

luzpaz commented Nov 14, 2024

@kieraneglin can you still reproduce on 5.0.1 ?

Closing in the meantime since there are 3 reports of users unable to reproduce anymore.

@luzpaz luzpaz closed this as completed Nov 14, 2024
@SanderBouwhuis
Copy link

SanderBouwhuis commented Nov 14, 2024

QBittorrent v5.0.1 (64-bit)
Qt: 6.7.3
Libtorrent: 2.0.11.0
Boost: 1.86.0
OpenSSL: 3.4.0
zlib: 1.3.1

I have about 400 torrents on both these two computers:
On Windows 10 Pro x64 : no memory issues after running for a few days with about 50 GB down and ? amount up.
On Windows 11 Pro x64 : no memory issues after running for 2 days with about 10 GB down and ? amount up.

@Dekumori
Copy link

I can confirm that after updating to the latest version, there are no longer any memory issues on my server. Thanks for the fix!

@m-aar-ten
Copy link

m-aar-ten commented Nov 15, 2024

@luzpaz

Closing in the meantime since there are 3 reports of users unable to reproduce anymore.

I am running 5.0.1 on Truenas Scale and it still happens, but instead of crashing and restarting the app, my whole server crashes.
I can confirm downgrading to 4.6.7 and migrating the installation has fixed all issues.

Edit: nvm, it seemed to have crashed again out of nowhere, if I boot it up again I seem to be having the same issues as before.

@HanabishiRecca
Copy link
Contributor

@m-aar-ten, try #21502 (comment)

@kieraneglin
Copy link
Author

@luzpaz I just checked and cannot repro on 5.0.1

@m-aar-ten
Copy link

m-aar-ten commented Nov 18, 2024

Another possible test: try to reset the client to default settings.
I.e. delete existing client config (make a backup if you want). The config location per platform is described in the wiki.

Also we need:

  1. Reproduction steps. Exactly what you do with the client, maybe some patterns you've noticed.
  2. Logs. Any info as much as possible.
  3. Crash reports, dumps and backtraces, if any.

P.S. If everyone going to flip the table and roll back to older versions, the problem would never be fixed.

So resetting my settings seemed to have fixed some of my issues, everything is running smoothly, downloading is working but the problem seems to be with rechecking for me.

I have narrowed it down to around 8 problematic torrents (coincidentally the largest ones), I rechecked the rest without any crash.

What is strange to me is that I have a 212gb torrent consisting of 10 large files, which is not causing me any issues.

But I had a 1TB torrent, consisting of many smaller (1-5gb) files which was causing my client to crash.

Also a 300gb torrent consisting of 2900 small files is also it to crash when selected for rechecking.
I attached my system logs here, sadly qBittorrent its doesnt give me any information in its logs.

#21842 (comment)

Edit: tried upgrading again from 4.6.4 which i got stable to 5.0.1 but that seemed to have messed everything up, back to square 1...

@HanabishiRecca
Copy link
Contributor

HanabishiRecca commented Nov 18, 2024

I had a 1TB torrent, consisting of many smaller (1-5gb) files which was causing my client to crash.

Is there any chance you are hitting #21011?

@m-aar-ten
Copy link

I had a 1TB torrent, consisting of many smaller (1-5gb) files which was causing my client to crash.

Is there any chance you are hitting #21011?

I will check it out once I have the time to get qbit stable and running again.

@m-aar-ten
Copy link

I had a 1TB torrent, consisting of many smaller (1-5gb) files which was causing my client to crash.

Is there any chance you are hitting #21011?

@HanabishiRecca you are correct, all of the torrents causing issues are exactly this limit., I got the torrents working again by removing them, adding them again and skipping recheck.

@HanabishiRecca
Copy link
Contributor

HanabishiRecca commented Nov 23, 2024

I got the torrents working again by removing them, adding them again and skipping recheck.

Note that 256 MiB piece size support was removed for libtorrent 1.2. This change is already in official qBittorrent 5.0.2 builds.
If you want to continue to use such torrents, consider switching to libtorrent 2.0 version.

@m-aar-ten
Copy link

I got the torrents working again by removing them, adding them again and skipping recheck.

Note that 256 MiB piece size support was removed for libtorrent 1.2. This change is already in official qBittorrent 5.0.2 builds.
If you want to continue to use such torrents, consider switching to libtorrent 2.0 version.

Thanks a lot, Ive removed these large torrents and memory usage has been super low and stable.
I'll use a 2nd client running 2.0 for them.

@blind-oracle
Copy link

blind-oracle commented Nov 27, 2024

I'm running 5.0.2 in a Docker container (lscr.io/linuxserver/qbittorrent:latest) and today I saw a 113GB(!) (out of 256GB) memory usage by qBittorrent... I have maybe 20-30 torrents seeding, nothing major. Disk access is set to memory mapped files, switched now back to POSIX to see if it makes a difference.

Will also check if WebUI makes it worse

@blind-oracle
Copy link

Yeah it seems that memory mapping was the culprit. After switching to POSIX the memory usage is stable and very low. Seeding speed does not look affected.

@HanabishiRecca
Copy link
Contributor

After switching to POSIX the memory usage is stable and very low.

You can try new Simple pread/pwrite mode.

@DeathStalker77
Copy link

Interesting! I had this issue on 4.6.7 standard!

--- DS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests