arbitrary reuse and check signatures in mocha BDD

[ssube]

I recently started using fast-check on a few projects, long having needed a good pbt library. The arbitraries and chain/map are very easy to use, even when generating complex data types, and I was able to find and fix a few bugs in my utility library.

The fc.assert syntax looked a little bit out of place with my existing mocha BDD tests, and I wanted to learn how fast-check worked internally, so I wrote a small wrapper (https://github.com/ssube/mocha-foam) that looks something like so:

import { expect } from 'chai';
import { integer } from 'fast-check';
import { over } from 'mocha-foam';

describe('example properties', () => {
  over('some numbers', integer(), (it) => {
    it('should be even', (n: number) => {
      return n % 2 === 0;
    });

    it('should be odd', async (n: number) => {
      expect(n % 2).to.equal(1);
    });
  });
});

The difference is the arbitrary at the suite level, using a wrapped it to call fc.check(fc.asyncProperty(strategy, test)) for each of the tests.

While writing this, I came across a few questions:

1. Is there a definitive way to tell from the check results, a RunDetails<T>, whether the property threw an error or returned falsy? I know the failed property indicates whether it failed in general, but the error property seems to be populated whether it returned or threw.
2. Sort of related to question 1, did anything come of 🐛 Bug: Reports the Error twice when its message contains something similar to a stack mochajs/mocha#3992 ? I see a related PR, fix(base): if message and stacktrace overlap remove duplicates mochajs/mocha#4399, using the Mocha utils. Is stackTraceFilter still the way to go?
3. Is there anything unusual or any problems with calling fc.check(fc.property(strategy, test)) multiple times with the same value for strategy? As in https://github.com/ssube/mocha-foam/blob/master/src/index.ts#L46, which occurs for each call to it.
4. The check for an Arbitrary<T> seems to expect Parameters<[T]>, changing from T to a single-element tuple containing T. That was indicating that examples for a tuple(integer(), integer()) should be wrapped into the somewhat confusing [[[1, 2]], [[3, 4]]]. Worked around in https://github.com/ssube/mocha-foam/blob/master/src/index.ts#L34. Is that an intentional part of the types, or a mistake in my wrapper?

[dubzzz]

Hey,

Here are some answers for your questions:

1. Is there a definitive way to tell from the check results, a RunDetails, whether the property threw an error or returned falsy? I know the failed property indicates whether it failed in general, but the error property seems to be populated whether it returned or threw.

Actually not really. As you said the error field will always be set in case of non-fulfill property. On fast-heck side return false or throwing is fully equivalent. The potential only thing you could do it checking for a precise error. You could probably do something like:

const out = fc.check(fc.property(...arbs, predicate));
if (out.failed) {
  if (out.error === 'Property failed by returning false') {
    // property returned false
  }
  else {
    // property crashed
  }
}
else {
  // success
}

But for the moment there is nothing better.

2. Sort of related to question 1, did anything come of Reports the Error twice when its message contains something similar to a stack 🐛 Bug: Reports the Error twice when its message contains something similar to a stack mochajs/mocha#3992 ? I see a related PR, fix(base): if message and stacktrace overlap remove duplicates fix(base): if message and stacktrace overlap remove duplicates mochajs/mocha#4399, using the Mocha utils. Is stackTraceFilter still the way to go?

Regarding this one I have no real news. fast-check now comes with a built-in way to customize the reporter to report errors in a custom way. You may have a look to: https://github.com/dubzzz/fast-check/blob/main/documentation/Tips.md#customize-the-reported-error

3. Is there anything unusual or any problems with calling fc.check(fc.property(strategy, test)) multiple times with the same value for strategy? As in https://github.com/ssube/mocha-foam/blob/master/src/index.ts#L46, which occurs for each call to it.

No there is not. Arbitraries should be state-less they can be called multiple times in a row without being impacted by previous calls*. So fell free to re-use them from one property to another.

*It is at least true for the arbitraries coming within fast-check.

4. The check for an Arbitrary seems to expect Parameters<[T]>, changing from T to a single-element tuple containing T. That was indicating that examples for a tuple(integer(), integer()) should be wrapped into the somewhat confusing [[[1, 2]], [[3, 4]]]. Worked around in https://github.com/ssube/mocha-foam/blob/master/src/index.ts#L34. Is that an intentional part of the types, or a mistake in my wrapper?

The input of check is supposed to be a property. The built-in property builder of fast-check builds tuples of all the inputs.

Nonetheless the typings of check seem to say check<Ts>(property: IProperty<Ts>, params?: Parameters<Ts>): RunDetails<Ts> (see https://github.com/dubzzz/fast-check/blob/main/src/check/runner/Runner.ts#L116), so nothing prevent to build a property not wrapping everything inside tuples and use check BUT the default one does so the tuple is expected.

[ssube]

Thank you for the quick and thorough reply, I think you've answered all of my questions.

1. For the error vs false, I had been looking for strings starting with an Error or AssertionError or similar string, but looking for the message fast-check uses does seem more reliable.
2. I have been using a custom error reporter, and will try adding the mocha functions to it, that will probably do the trick.
3. Excellent
4. I had not noticed the tuple was coming from the property, but that all makes sense, I see how it is occurring. It's not a problem, necessarily, and easy enough to wrap if needed.