Out of memory issue - Docker 137 exit code

[benoit74]

We have some tasks which have failed with out of memory issues:

• https://farm.openzim.org/pipeline/4ff2a566-eef3-4fc9-9b14-b575e1c97140/debug
• https://farm.openzim.org/pipeline/14e6e9a5-29e6-4131-a86d-baac839315da/debug
• https://farm.openzim.org/pipeline/216217c6-8997-4d9e-95cf-ce24b02d87f7/debug

[benoit74]

Analyzing https://farm.openzim.org/recipes/libretexts.org_en_biz subsequent failures, it looks like problem occurs somewhere around page ID 89303 of biz.libretexts.org (https://biz.libretexts.org/Bookshelves/Business/Advanced_Business/Construction_Contracting_-_Business_and_Legal_Principles/01%3A_Chapters/1.06%3A_Labor_Agreements). Its parent is page ID 89296.

I hence ran a scrape of biz.libretexts.org with page ID 89296 and tried many different settings / hacks.

The conclusions (I did not recorded all attempts, I'm lazy) are:

• only the page 89296 and its children consume about 14G of RSS with "default" settings
• the problem is not the fact that we pass asset "in-memory" to the creator
• the problem is not in S3 cache operations (if we disable the cache, we have the same memory consumption)
• the problem is gone once S3 cache is properly populated
• the problem is gone when we remove conversion to WebP and optimization (just downloading assets and passing them to the ZIM)
• the problem is less visible when we do only conversion to WebP and remove optimization (~5G memory consumed)
• the problem is less visible when we reduce the number of assets workers (less images manipulated in parallel)
• there are some very huge images on these pages like https://biz.libretexts.org/@api/deki/files/38713/6-5.png?revision=1 (5485x2848 for a "simple" flowchart 🤦🏻‍♂️)
• resizing images to 720px height at most (HD ready 😅, with quick and dirty code making many assumptions about image orientation for instance) before conversion to WebP and optimization allow to reduce memory consumption to 2.3G
  • reducing number of assets workers helps even further (little less than 1G when 2 workers instead of 10)
  • this is my proposed solution, PR under way ^^

For the record, a very useful solution to observe maximum memory consumption: /usr/bin/time -v <your_command> (do not use just time, which is overridden in many shells, use /usr/bin/time).