Best-practice for guarding against infinite-looping arbitraries?

[amccarthy1]

Hey folks, we've started to see this issue crop up more and more in which a buggy .filter results in an infinite loop, and seemingly none of the "Stop executing after x time" will stop it, since the loop happens before the predicate starts executing. This leads to difficult-to-debug tests because we get no logging output, and you have to see that the job is hanging and kill it manually, due to limitations with test runners not being able to interrupt sync infinite loops.

I've seen some recommendations around using fc.pre instead, but is there any way to add safeguards to prevent infinite loops while generating arbitrary values? I considered opening a bug report about this, but wanted to check here first.

Here's an example of an arbitrary that infinite loops, and can't be killed by interruptAfterTimeLimit

  it('Should interrupt the test run after a short time', async () => {
    fc.assert(
      fc.property(
        fc.nat().filter(() => false),
        (_value) => {}
      ),
      { interruptAfterTimeLimit: 1000 }
    );
  });

(this arbitrary is obviously wrong, but there are some invocations that can be surprisingly hard to spot)

Are there any configs that will cause this type of test not to loop indefinitely?

[dubzzz]

At the moment, the only built-in thing that can protect you against filtering leading to an infinite loop is fc.pre.

For now, I think the only trick that could make it for filter would be to do something like:

function buildStoppeable(filter) {
  let consecutiveErrors = 0;
  return function stoppeableFilter(...args) {
    const out = filter(...args);
    if (out) {
      consecutiveErrors = 0;
      return out;
    }
    if (++consecutiveErrors > 100) throw new Error('To many errors!');
    return out;
  }
}

it('Should interrupt the test run after a short time', async () => {
  fc.assert(
    fc.property(
      fc.nat().filter(buildStoppeable(() => false)),
      (_value) => {}
    ),
    { interruptAfterTimeLimit: 1000 }
  );
});

And if you want to make it the default for filter (note: this prototype-based version might break on next minors as it plays with altering internals):

const originalFilter = fc.Arbitrary.prototype.filter;
fc.Arbitrary.prototype.filter = function (...args) {
  const f = buildStoppeable(args[0]);
  return originalFilter.call(this, f, ...args.slice(1));
}

Side-note:

For the moment, anything around time limit, timeout and similar is only related to things related to the execution of the predicates. They are controlled outside of the arbitraries and can only stop synchronous code.

[dubzzz]

So far, I logged #4660 to track it and think of it in my next iterations.