[WIP] more extensive IAVL v2 testing

v2/README.md

@@ -1 +1,104 @@
 # IAVL v2
+
+https://www.youtube.com/watch?v=keV22tP8nks
+
+## Node Key Format
+
+Node keys are 12-byte arrays with the following structure:
+
+- 8 bytes for version (big-endian uint64)
+- 4 bytes for sequence (big-endian uint32)
+    - sequence numbers are unique within a version and incremented for each new node
+    - leaf node sequences have the high bit set (>= 0x80000000) to distinguish them from internal nodes
+    - leaf node sequence numbers are assigned both for insertion and deletions, in the order of operations (so that they
+      can be used to reconstruct the tree)
+
+## Insertion Algorithm
+
+## Database Structure
+
+* leaf nodes and internal nodes are stored in separate tables
+* leaf nodes are stored in the `leaf` table
+* internal nodes are stored in `tree_{version}` tables, where `{version}` is the version of the tree, but these trees
+  are only present for versions that are checkpoints (TODO: verify this)
+
+### Database Files
+
+#### `changelog.sqlite`
+
+Tables:
+
+* `latest`
+* `leaf`
+* `leaf_delete`
+* `leaf_orphan`
+* `snapshot_{version}` (for each snapshot version)
+
+#### `tree.sqlite`
+
+Tables:
+
+* `root`
+* `tree_{version}` (for each checkpoint version)
+* `orphan`
+
+### `root` table
+
+Columns:
+
+- `version`: int, the version of the tree
+- `node_version`: int, the version of the node
+- `node_sequence`: int, the sequence number of the node (unique within the version)
+- `bytes`: blob, encoding?? TODO
+- `checkpoint`: bool, whether this version is a checkpoint
+- `PRIMARY KEY (version)`: ensures each version is unique
+
+### `tree_{version}` and `leaf` tables
+
+Columns:
+
+- `version`: int, the version of the tree
+- `sequence`: int, the sequence number of the leaf node insertion (unique within the version)
+- `bytes`: blob, see [encoding below](#node-bytes-encoding)
+- `orphaned`: bool, whether the leaf node is orphaned (TODO)
+
+### `leaf_delete` table
+
+Columns:
+
+- version: int, the version of the tree
+- sequence: int, the sequence number of the leaf node deletion (unique within the version)
+- key: bytes, the key of the leaf node that was deleted
+
+### `leaf` and `leaf_delete` tables functions as changelog
+
+The `leaf` and `leaf_delete` tables function as a changelog for the tree.
+Given a tree at a given checkpoint (or genesis), the entries in the `leaf` and `leaf_delete` tables can be in
+sequence order to reconstruct the tree at any target version.
+Recall that the hash of a tree is dependent on insertion order.
+Because sequence numbers are assigned in order of insertion/deletion within a version,
+they can be used to accurately reconstruct the tree at any point in time.
+
+### Node `bytes` encoding
+
+In `leaf` table and `tree_{version}` tables, the `bytes` field is encoded as follows:
+
+* `height` varint, `int8` max range
+* `size` varint
+* `key` bytes (varint length prefixed)
+* `hash` bytes (varint length prefixed)
+* if leaf node:
+    * `value` bytes (varint length prefixed)
+* else (internal node):
+    * `leftNodeKey` (varint length prefixed but always 12 bytes)
+    * `rightNodeKey` (varint length prefixed but always 12 bytes)
+
+## Loading a Version
+
+* get list of versions which are marked as checkpoints
+* load the first checkpoint before or at the target version
+* replay all `leaf` and `leaf_delete` entries from that checkpoint version up to the target version
+
+## Eviction
+
+When a tree is checkpointed, nodes at a height >= eviction height are removed from memory.

v2/go.mod

@@ -12,24 +12,46 @@ require (
 	github.com/prometheus/client_golang v1.21.1
 	github.com/spf13/cobra v1.9.1
 	github.com/stretchr/testify v1.10.0
-	golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
+	golang.org/x/exp v0.0.0-20231006140011-7918f672742d
 )
 
 require (
+	cosmossdk.io/log v1.6.0
+	github.com/cosmos/iavl v1.3.5
+	pgregory.net/rapid v1.2.0
+)
+
+require (
+	cosmossdk.io/core v0.12.1-0.20240725072823-6a2d039e1212 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/bytedance/sonic v1.13.1 // indirect
+	github.com/bytedance/sonic/loader v0.2.4 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/cloudwego/base64x v0.1.5 // indirect
+	github.com/cosmos/gogoproto v1.5.0 // indirect
+	github.com/cosmos/ics23/go v0.10.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/snappy v0.0.4 // indirect
+	github.com/google/btree v1.1.2 // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/klauspost/compress v1.18.0 // indirect
-	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.10 // indirect
+	github.com/mattn/go-colorable v0.1.14 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/prometheus/client_model v0.6.1 // indirect
 	github.com/prometheus/common v0.62.0 // indirect
 	github.com/prometheus/procfs v0.15.1 // indirect
-	github.com/rs/zerolog v1.33.0 // indirect
+	github.com/rs/zerolog v1.34.0 // indirect
 	github.com/spf13/pflag v1.0.6 // indirect
+	github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect
+	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
+	golang.org/x/arch v0.15.0 // indirect
+	golang.org/x/crypto v0.23.0 // indirect
 	golang.org/x/sys v0.31.0 // indirect
 	google.golang.org/protobuf v1.36.6 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect