Description of Changes

This PR changes the structure of the datastore to insert via BSATN rather than a PV. But this isn't entirely true, we just use the fast-path for BSATN -> BFLATN but in the general case we still do BSATN -> PV -> BFLATN. Later, I'll fully remove the temporary step. One of the other required changes here is that we write and generate sequence values directly in BFLATN.

Fixes #2017.

Perf numbers on master

2025-01-03T09:30:37.776998Z  INFO: : Timing span "update_positions_by_id": 1.969988054s
2025-01-03T09:30:43.829835Z  INFO: : Timing span "update_positions_by_collect": 1.518314839s

with this PR:

2025-01-03T09:35:54.415502Z  INFO: : Timing span "update_positions_by_id": 1.64947372s
2025-01-03T09:36:00.880068Z  INFO: : Timing span "update_positions_by_collect": 1.26073871s

That is, this removes ~0.4516s and 0.3887s from the benchmarks respectively.

Flamegraph of InstanceEnv::insert:

Full flamegraph: https://flamegraph.com/share/14b571da-c9ba-11ef-9832-26c3e5347170

The next step after this PR is to avoid the pointer map in case there is a unique index.
After that, the next step is to add an update ABI.

API and ABI breaking changes

None

Expected complexity level and risk

4 -- limited scope, but lots of unsafe code and complicated logic.

Testing

The internal spacetimedb_table tests still use the old table.insert, so the new path. To compensate for this, without duplicating tests, a new proptest insert_bsatn_same_as_pv is added asserting that the result and side-effects of inserting via PV and BSATN are the same. Moreover, both insert paths try to share as much code as possible to improve test coverage. The higher level tests, starting with MutTxId now use the new path. Over time, we can replace the remaining old paths with the new and then move all tests to the new as well.

Reviewer notes

I would recommend reviewing in this order:

1. Start with spacetimedb_table
2. Move on to locking_tx_datastore
3. Move on to InstanceEnv
4. Review all the other boring changes to files like estimation.rs that were unfortunately necessary.

Review notes for Tyler

In traits.rs the following changes:

• insert_mut_tx changes from:

  fn insert_mut_tx<'a>(..., row: ProductValue) -> Result<(AlgebraicValue, RowRef<'a>)>;

  to:

  fn insert_mut_tx<'a>(..., row: &[u8]) -> Result<(ColList, RowRef<'a>)>;