Determinstic tests

[SRachamim]

Hey, thanks you all for this awesome library! Countless are the bugs we eliminate by using fast-check! Especially in a functional TypeScript environment.

One issue we do have is with the indeterministic spirit of fast-check. We want our tests to be determinstic. They shouldn't block a PR remotely if the same commit passed locally. Unfortunately, from time to time, we see different test coverage and surprising failing tests only remotely. Surely, we can reproduce it locally with the seed, but still we don't like surprises later in the workflow (or even in different PRs on unrelated changes).

How can we cope with it? Is there a way to run all possible end-cases all the time?

[dubzzz]

If you want your CI, to be deterministic you must setup fast-check to only use one precise seed by using fc.configureGlobal({ seed: 48 }). But I tend to discourage such usage as it would mean not benefit from the strength of property based tests which is to find deeply hidden bugs that might be leveraged by attackers or be the origin of nasty bugs. If you do use such thing in your CI, I'd highly recommend to also have a seed-free version to run on a regular basis.

[dubzzz]

More concerning global settings at https://github.com/dubzzz/fast-check/blob/main/documentation/Tips.md#setup-global-settings

[dubzzz]

By the way if you use fast-check within jest, you might be interested into @fast-check/jest as it leverages the new --seed cli option of jest and connect it to fast-check automatically.

[SRachamim]

Thanks for your answer! Still hope there's a way to run all possible tests, even if they'll take too long.

[dubzzz]

Still hope there's a way to run all possible tests, even if they'll take too long.

Unfortunately, probably something that will take an infinite amount of time. There is an infinite amount of possible strings, generating all of them takes an infinite amount of time...

[dubzzz]

Just marked the discussion as solved, but please feel free to reopen it if there are still pending issues or questions.

[SRachamim]

Still hope there's a way to run all possible tests, even if they'll take too long.

Unfortunately, probably something that will take an infinite amount of time. There is an infinite amount of possible strings, generating all of them takes an infinite amount of time...

Oh well, at least the edge cases. For example, the __proto__ string is an important edge case I always want to check in all tests.