Reuse chunks from the same download if duplicated

[Roman513]

Drastically decrease memory usage by reusing chunk parts from already downloaded files.
See derrod#17

Usually we do not need even 1Gb of shared memory with this patch.

This is the same approach and roughly the same code as we have for reusing chunks of old game files on update couple lines above.

This is a copy of PR against upstream derrod#683 to speed up delivery of solution for Heroic users

Should help with issues like Heroic-Games-Launcher/HeroicGamesLauncher#2732, Heroic-Games-Launcher/HeroicGamesLauncher#4054, derrod#149,
derrod#151

Personally I was able to download "The Talos Principle 2" with less than 300 Mb of RAM cache usage all the time. Before this patch, about 12Gb was required.

Testing and reviews welcome.

[Gurjiwan]

I was trying to install a game on heroic and had a memory issue as well.
Heroic games launcher uses shared memory cache as 2048.0 MiB for initial run and 5000.0 MiB in case of failure but both were not enough to download STASIS : BONE TOTEM game from epic game store.

Legendary: 0.20.36 Urban Flight (Heroic)

MemoryError: Current shared memory cache is smaller than required: 5000.0 MiB < 16312.0 MiB. Try running legendary with "--enable-reordering" to reduce memory usage, or use "--max-shared-memory 16344" to increase the limit.

Issue reference: Heroic-Games-Launcher/HeroicGamesLauncher#4231

Please confirm if it is same issue as with the games tested by you and use it as a test case in further development if possible.

[Roman513]

@Gurjiwan you can test if this patch helps with your issue using custom Legendary build with heroic from a patch branch. The most probably it is.

[Gurjiwan]

@Gurjiwan you can test if this patch helps with your issue using custom Legendary build with heroic from a patch branch. The most probably it is.

Yes the download started normally after this patch. i didn't completely download the game since i had already used a silly workaround as mentioned in the issue reference above. I just tested with and without the custom legendary and checked what is logged when i hit download post moving my old install files.

Without this pull, it required 16gb of my ram as shared memory and still fails a couple of times but this worked smoothly.

Steps taken to get the custom legendary:

1. git clone https://github.com/Heroic-Games-Launcher/legendary.git
2. git fetch origin pull/11/head:pr/11
3. git checkout pr/11

i built and installed it and added this to my heroic and no more shared memory cache issue. It looked like it was completed in a single run too so probably took less than 2 gb as well.