Potential virtual memory leak #5189
Comments
|
Something I forgot to mention - it lists as trying to consume 263G as soon as the process starts -- there's no waiting time. |
|
So this is VIRT memory usage. This is something that the underlying technology (.net) does. Here’s an explanation dotnet/runtime#9199 but this shouldn’t really be an issue or causing any issues |
|
Wouldn't have reported it unless it was having issues 😉 Either way, the cron job has kept it alive at the cost of bloating my logs which is tolerable, and I can understand if there's no real fix here if the issue is with the underlying dependency itself. I'm good if you want to close this as can't/won't fix, figured you should be aware at the very least that the process is getting killed by the OOM watch dog on some platforms. |
|
spent a good bit writing scripts to separate all my logs into separate files under /var/log - it looks like it might have been a database issue, so my earlier assumption that the issue was the OOM protection was likely incorrect. cron seems to run before systemctl, and as a result the mysql service hadn't started. ombi tries to hook into it, and closes itself as a result.
going to wait a few days to make sure there wasn't something else crashing it before closing this out if it is a timing issue, is it possible to put ombi into time limited sleep loop to allow mysql to load? |
Summary
Overview
Ombi seems to be running into an issue where it tries to consume more virtual memory than is physically possible (263G). As a result, it keeps getting flagged by the system resource manager as needing to die, and it gets killed. I've implemented a hotfix via cron, but it's messy and doesn't solve the base issue.
The odd part is that the system itself isn't burdened - memory and CPU are all nominal. For whatever reason, linux thinks that Ombi is chewing up 263G of memory.
Hotfix
/etc/cron.d\# touch oom_disable/ect/cron.d\# echo '*/1 * * * * root pgrep -f "Ombi\ --storage\ [long storage path]" | while read PID; do echo -17 > /proc/$PID/oom_adj; done'This just persistently keeps the process from being killed. Not the best solution.
Environment setup
su daemon-ombi -s /bin/bash -c 'cd /home/daemon-ombi/bin && /home/daemon-ombi/bin/Ombi --storage "/media/[long storage path]" --host "http://0.0.0.0:40011" --baseurl "/ombi"' &Relevant Error logs
2024-11-22 14:49:55.180 -05:00 [INF] Default Quartz.NET properties loaded from embedded resource file2024-11-22 14:49:55.255 -05:00 [INF] Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl2024-11-22 14:49:55.255 -05:00 [INF] Quartz Scheduler created2024-11-22 14:49:55.255 -05:00 [INF] RAMJobStore initialized.2024-11-22 14:49:55.256 -05:00 [INF] Quartz Scheduler 3.6.2.0 - 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED' initialized2024-11-22 14:49:55.256 -05:00 [INF] Using thread pool 'Quartz.Simpl.DefaultThreadPool', size: 102024-11-22 14:49:55.256 -05:00 [INF] Using job store 'Quartz.Simpl.RAMJobStore', supports persistence: False, clustered: False2024-11-22 14:49:55.257 -05:00 [INF] JobFactory set to: Ombi.Schedule.IoCJobFactory2024-11-22 14:49:55.304 -05:00 [INF] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.Relevant Picture
Ombi Version
4.44.1
What platform(s) does this occur on?
Linux
What database are you using?
MySQL
Relevant log output
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5910] (rev 05) Subsystem: ASUSTeK Computer Inc. Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [1043:1bb0] Kernel driver in use: skl_uncore 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05) Kernel driver in use: pcieport 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:591b] (rev 04) DeviceName: Onboard IGD Subsystem: ASUSTeK Computer Inc. HD Graphics 630 [1043:1bb0] Kernel driver in use: i915 Kernel modules: i915 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911] Subsystem: ASUSTeK Computer Inc. Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [1043:1bb0] 00:14.0 USB controller [0c03]: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller [8086:a12f] (rev 31) Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller [1043:201f] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:14.2 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem [8086:a131] (rev 31) Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Thermal Subsystem [1043:1bb0] Kernel driver in use: intel_pch_thermal Kernel modules: intel_pch_thermal 00:15.0 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Serial IO I2C Controller #0 [8086:a160] (rev 31) Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Serial IO I2C Controller [1043:1bb0] Kernel driver in use: intel-lpss Kernel modules: intel_lpss_pci 00:16.0 Communication controller [0780]: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 [8086:a13a] (rev 31) Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family MEI Controller [1043:1bb0] Kernel driver in use: mei_me Kernel modules: mei_me 00:17.0 RAID bus controller [0104]: Intel Corporation 82801 Mobile SATA Controller [RAID mode] [8086:282a] (rev 31) Subsystem: ASUSTeK Computer Inc. 82801 Mobile SATA Controller [RAID mode] [1043:1bb0] Kernel driver in use: ahci Kernel modules: ahci 00:1c.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #5 [8086:a114] (rev f1) Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #6 [8086:a115] (rev f1) Kernel driver in use: pcieport 00:1c.6 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 [8086:a116] (rev f1) Kernel driver in use: pcieport 00:1f.0 ISA bridge [0601]: Intel Corporation HM175 Chipset LPC/eSPI Controller [8086:a152] (rev 31) Subsystem: ASUSTeK Computer Inc. HM175 Chipset LPC/eSPI Controller [1043:1bb0] 00:1f.2 Memory controller [0580]: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller [8086:a121] (rev 31) Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family Power Management Controller [1043:1bb0] 00:1f.3 Audio device [0403]: Intel Corporation CM238 HD Audio Controller [8086:a171] (rev 31) Subsystem: ASUSTeK Computer Inc. CM238 HD Audio Controller [1043:1bb0] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1f.4 SMBus [0c05]: Intel Corporation 100 Series/C230 Series Chipset Family SMBus [8086:a123] (rev 31) Subsystem: ASUSTeK Computer Inc. 100 Series/C230 Series Chipset Family SMBus [1043:1bb0] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 01:00.0 3D controller [0302]: NVIDIA Corporation GP107M [GeForce GTX 1050 Mobile] [10de:1c8d] (rev a1) Subsystem: ASUSTeK Computer Inc. GP107M [GeForce GTX 1050 Mobile] [1043:1bb0] Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) Subsystem: ASUSTeK Computer Inc. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [1043:200f] Kernel driver in use: r8169 Kernel modules: r8169 03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5229 PCI Express Card Reader [10ec:5229] (rev 01) Subsystem: ASUSTeK Computer Inc. RTS5229 PCI Express Card Reader [1043:202f] Kernel driver in use: rtsx_pci Kernel modules: rtsx_pci 04:00.0 Network controller [0280]: Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78) Subsystem: Intel Corporation Dual Band Wireless-AC 8265 [8086:0010] Kernel driver in use: iwlwifi Kernel modules: iwlwifiThe text was updated successfully, but these errors were encountered: