[Feature]: Allow pseudorandom test distribution in sharding

[gerardaz]

🚀 Feature Request

Create a config in Playwright to allocate fullyParallel tests in some sequence that more "randomly" distributes tests among shards.

I know the shards must determine their tests independently of each other, so it can't really be random. But this could be something like round robin, or by performing a modulus on the testID.

Example

parallelDistribution?: "chunk" | "round-robin" | "modulus"

Or something like this, where the default behaviour is "chunk" if unspecified, and does nothing if not fullyParallel.

Doesn't need to implement all of these – all I would really like is something that does not inherently make adjacent tests more likely to be in the same shard. I'm okay with some imbalance

Motivation

This assumes that even fullyParallel tests are distributed by chunking. Like tests 0 to n, n+1 to 2n, 2n+1 to 3n etc. If I'm incorrect in this understanding please let me know.

When sharding with fullyParallel: true tests in the same file can end up in different shards, allowing better balancing (e.g. if some files have more tests than others).

But if you have some tests that are very slow – e.g. if you were testing a platform that does video transcoding, this would be slow to verify the result – there is a very high chance these tests will appear adjacent to each other in the list of tests.

As a result, these slow tests have a much higher chance of ending up in the same shard.

Right now I run a suite of 300 tests in 50 shards. Most of the tests take only 10s or less to run. But there is a particular group of tests testing a slow behaviour, which take about 5m each, and nearly always end up in the same shard as each other, even with fullyParallel: true