Split explorer storage in stable and unstable #3205
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
the important part here is that we only need to remove from the beginning, because this is a blockchain, and we are going to be undoing blocks from the back.
doing it in set_tip doesn't work, because we do that *after* applying the block, and then we won't have the stable_block in the tip's branch also, prevents some concurrency issues
mostly to account for the fact that at some point data won't be entirely in memory, so we need to take epoch_stability_depth and utxo inputs into account
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Intermediate step to introduce use a non-volatile database for the explorer. It doesn't change any behavior by itself, so it could be merged individually, it's mostly to prepare things.
About
This PR
apply_blockoperation from blocks moved to the stable storage, in any incoming multiverse State.The last point requires some explanation. Currently the explorer has a multiverse of
State, which is entirely a persistent data structure. This means we have an immutable version of the indexes for each block that we have, and there is no way to simply forget things in there, but if we don't remove things that we move to the stable storage, then there will be no point in doing it.Because of this, I took the approach of applying delayed deletes. Each time we add a
State, we also unapply a block from the past (the one epoch_stability_depth blocks behind), before inserting.See the following graph of the multiverse for a graphical illustration.
For the notation:
Statein the multiverse, which in this case represents a snapshot of the explorer index. This is what's used to answer queries by providing a branch id through the api.Important: this is just partial, although it can be merged as it is, this is still missing the
undooperations for the vote tally indices (mostly because I don't quite understand that one entirely).Note: I wrote something about this in #2759, but is better to read this first, because I still need to update that one with more up-to-date thoughts.