Add cut-off mechanism for cell relocation to improve performance

[asonnino]

Problem

Currently, the cell relocation process processes all cells regardless of their deletion ratio or size, which can lead to inefficient processing of cells that have minimal benefit from relocation.

Proposed Solutions

Option 1: Deletion Ratio Cut-off

Add a cut-off at the cell level. If a cell has very few entries to remove (e.g., < 10% deletion ratio), skip relocating the entire cell. We can count bloom filter misses as a proxy for deletions:

let mut likely_removals = 0;
let total_entries = index.iter().count();

if let Some(bloom) = bloom_filters.get(&cell_ref.keyspace_desc.id()) {
    for (key, position) in index.iter() {
        let reduced_key = cell_ref.keyspace_desc.reduce_key(key);
        if !bloom.contains(&LargeTable::bloom_key(&reduced_key, *position)) {
            likely_removals += 1;
        }
    }

    let removal_ratio = likely_removals as f32 / total_entries as f32;
    if removal_ratio < 0.1 {
        return Ok(context); // Skip cell
    }
}

Option 2: Cell Size Threshold

Skip cells smaller than a certain threshold size:

if index.iter().count() < CELL_SIZE_THRESHOLD {
    // Skip this cell entirely
    return Ok(context);
}

[asonnino]

Note on gc_watermark calculation

When implementing the cut-off mechanism, we need to carefully consider how skipping cells affects the gc_watermark calculation in save_progress() (lines 149-160).

Currently, the gc_watermark for cell-based relocation uses:

std::cmp::min(
    cell_watermark.highest_wal_position,
    cell_watermark.upper_limit,
)

If we skip cells based on deletion ratio or size threshold, we need to ensure:

1. Skipped cells' WAL positions are not included in highest_wal_position
2. Or we track skipped cells separately to avoid premature GC of their WAL entries

Otherwise, we risk GC'ing WAL segments that still contain entries from skipped cells, which could lead to data loss if those cells need to be processed later.

[asonnino]

@zablotchi we may need to solve this before starting benchmarks

[asonnino]

@andll @phoenix-o Any opinion about this one?

[andll]

I think the way index-based relocation should work is to determine certain cut-off point where it is going to (relocation_target), and then only relocate wal positions below such relocation target. And after the relocation is done it can set the gc boundary to that target.

How we determine this can be somewhat usecase specific, we can start with something like relocating last 10% of the wal for example.