Consider Alternatives To DashMap

[XAMPPRocky]

(See: xacrimon/dashmap#285)

While doing loadtests and benchmarking, I've noticed a significant amount of time is spent dropping Arcs from read or write references to DashMap. DashMap is great for concurrent access to multiple shards, but our usage is much more "access everything", where we iterate over the whole map, this causes a tonne of extra overhead that I worked around in a commit by just cloning the endpoints into a separate vec as a cache.

We should consider alternatives to dashmap that are much more designed for our usage patterns.

quilkin/src/net/cluster.rs

Line 62 in 0a2b4e6

pub struct ClusterMap(DashMap<Option<Locality>, BTreeSet<Endpoint>>);

[xacrimon]

See xacrimon/dashmap#285 (comment) for some alternative suggestions based on your usage, though I only briefly skimmed over how it's used in quilkin. Should be an easy swap, not a lot of work to do.

[xacrimon]

My suggestion notably makes other significant optimizations trivial. For example your endpoint iteration spends a lot of time collecting a vector of endpoints, just to return it as an iterator. This may be useful under dashmap to release locks earlier but with an ArcSwap<HashMap> you can simply use load the arcswap into a guard and use the ::map method to "project" a Guard<HashMap> into a Guard<impl Iterator>, iterating and flatmapping on the maps entries like currently , and since readers no longer block writers from swapping out the value, while the one you have stays the same, there's no need to perform any intermediate collections, looks like this is #828.

Really on hot paths any kind of iteration + collection is probably a significant bottleneck. Unless you need sub millisecond update latency, using this clone swap update scheme should be fine, even with many thousands of map entries, should allow you to eliminate all of the intermediate allocations introduced in #855. Do note that cloning the btreesets might be a potential perf bottleneck on updates.

What you want to do to resolve that depends on if you have many endpoints comparatively to locality entries or not. I would consider and benchmark flattening them into a Vec of endpoints keyed by locality, always storing them sorted allowing you to use Vec::binary_search_* for "set lookups" if you don't have extremely large amounts of them per locality. If update performance is still prohibitive, I'd solve that by wrapping each key value (as either btreeset or vector, but my hunch is that vector will be better for your usecase) in another arcswap, so that they're not deep-cloned when updating clustermap.