Replace BlockTreeDb's leveldb-based storage with flat data files #165
+7,199
−99
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: David Gumberg <[email protected]> Co-authored-by: Lőrinc <[email protected]>
Co-authored-by: fanquake <[email protected]>
This is still available in the testing repo: https://github.com/bitcoin-dev-tools/benchcoin-testing
this just creates needless rebasing. Remove it.
sedited
force-pushed
the
blocktreestore_bench
branch
3 times, most recently
from
f51c2e1 to
d9c6c8b
Compare
|
📊 Benchmark results for this run (14778586774) will be available at: https://bitcoin-dev-tools.github.io/benchcoin/results/pr-165/14778586774/index.html after the github pages "build and deployment" action has completed. |
|
📊 Benchmark results for this run (14792561147) will be available at: https://bitcoin-dev-tools.github.io/benchcoin/results/pr-165/14792561147/index.html after the github pages "build and deployment" action has completed. |
The BlockTreeStore introduces a new data format for storing block indexes and headers on disk. The class is very similar to the existing CBlockTreeDB, which stores the same data in a leveldb database. Unlike CBlockTreeDB, the data stored through the BlockTreeStore is directly serialized and written to flat .dat files. The storage schema introduced is simple. It relies on the assumption that no entry is ever deleted and that no duplicate entries are written. These assumptions hold for the current users of CBlockTreeDB. In order to efficiently update a CBlockIndex entry in the store, a new field is added to the class that tracks its position in the file. New serialization wrappers are added for both the CBlockIndex and CBlockFileInfo classes to avoid serializing integers as VARINT. Using VARINT encoding would make updating these fields impossible, since changing them might overwrite existing entries in the file. This commit is part of a series to replace the leveldb database currently used for storing block indexes and headers with a flat file storage. This is motivated by the kernel library, where the usage of leveldb is a limiting factor to its future use cases. It also offers better performance and has a smaller on-disk footprint, though this is mostly negligible in the grand scheme of things.
This commit is part of a series to replace the leveldb database currently used for storing block indexes and headers with a flat file storage. This is motivated by the kernel library, where the usage of leveldb is a limiting factor to its future use cases. It also offers better performance and has a smaller on-disk footprint, though this is mostly negligible in the grand scheme of things.
This hooks up the newly introduce BlockTreeStore class to the actual codebase. It also adds a migration function to migrate old leveldb block indexes to the new format on startup. The migration first reads from leveldb (blocks/index), and writes it to a BlockTreeStore in a separate migration directory (blocks/migration). Once done, the original directory (blocks/index) is deleted and the migration directory renamed to the original name. This commit is part of a series to replace the leveldb database currently used for storing block indexes and headers with a flat file storage. This is motivated by the kernel library, where the usage of leveldb is a limiting factor to its future use cases. It also offers better performance and has a smaller on-disk footprint, though this is mostly negligible in the grand scheme of things.
These are no longer needed after the migration to the new BlockTreeStore. This commit is part of a series to replace the leveldb database currently used for storing block indexes and headers with a flat file storage. This is motivated by the kernel library, where the usage of leveldb is a limiting factor to its future use cases. It also offers better performance and has a smaller on-disk footprint, though this is mostly negligible in the grand scheme of things.
Adds constants for pre-allocating the file size of the header storage file in the BlockTreeStore. The chosen constants leave a bit of extra space beyond the actual requirement. They may be updated on every release, though it is also not a strict requirement to do so. This commit is part of a series to replace the leveldb database currently used for storing block indexes and headers with a flat file storage. This is motivated by the kernel library, where the usage of leveldb is a limiting factor to its future use cases. It also offers better performance and has a smaller on-disk footprint, though this is mostly negligible in the grand scheme of things.
This is not called by anything anymore, so just remove it. This commit is part of a series to replace the leveldb database currently used for storing block indexes and headers with a flat file storage. This is motivated by the kernel library, where the usage of leveldb is a limiting factor to its future use cases. It also offers better performance and has a smaller on-disk footprint, though this is mostly negligible in the grand scheme of things.
Also make flags based on file existence, instead of complicated boolean fields. This makes the operations atomic.
sedited
force-pushed
the
blocktreestore_bench
branch
from
fbd53d3 to
2291916
Compare
|
📊 Benchmark results for this run (15522915985) will be available at: https://bitcoin-dev-tools.github.io/benchcoin/results/pr-165/15522915985/index.html after the github pages "build and deployment" action has completed. |
willcl-ark
force-pushed
the
master
branch
7 times, most recently
from
d216dc5 to
7f7e173
Compare
willcl-ark
force-pushed
the
master
branch
6 times, most recently
from
5a6cd85 to
d9d7b46
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is just intended as a performance regression check. Not using leveldb for storage of the headers has big implications for the kernel project: It allows parallel block reading from another application while a bitcoind instance is running, opening the doors for potential novel indexing and wallet scanning approaches.