feat(filestore): add mmap reader option #665
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
I have a custom IPFS implementation based on boxo, which uses
FileManager
on Windows because I need to store files directly rather than file blocks. The pull logic is roughly as follows:ProtoNode
tree.RawNode
file blocks and write them to the system. Meanwhile, this Windows node also communicates and synchronizes with other nodes.When hundreds of Windows nodes are working simultaneously, I noticed that the system's memory consumption becomes very high, and the memory is immediately released once the file download is complete. Interestingly, the memory consumption is not by the program itself but by the operating system. After some investigation, I found the root cause: Windows caches the read file blocks (even if the *os.File is already closed).
This can be reproduced with the following code:
Solution
Using mmap can solve this problem (implemented with
CreateFileMapping
on Windows). To maintain backward compatibility, aWithMMapReader
option has been added toFileManager
. Enabling this option on Windows can prevent excessive memory consumption.The downside is that it relies on the exp package, but it only changes from an indirect dependency to a direct dependency. Alternatively, the code from this package can be copied directly into the project.