Wanted: Some way to reproduce results

[lilyball]

I just had a test failure (where I couldn't see the counterexample, see #115), but I can't reproduce it. It would be great if test failures would print the seed and if there was a trivial way of plugging that seed back into the test to reproduce that case.

[lilyball]

Well, I guess it needs both seed and size. And I just found CheckerArguments.replay which lets me specify the seed/size for the property, but since the test failure didn't print it, I don't know what it was.

[CodaFi]

Verbose output seems the best place to put that when people ask for it.

[lilyball]

My concern is that if it doesn't show me the seed when it fails, if it's a rare failure it's already too late to go back and request verbose output. For example, in the failure I had that prompted this ticket, I still haven't reproduce it. I'm about to bump up maxAllowableSuccessfulTests to some high number in the hopes of seeing it again.

[CodaFi]

If you convert it to an existential, SwiftCheck will bombard it for you trying to search for a passing value.

[lilyball]

I admit to not really being sure how to do that. I found the exists function, but presumably just replacing forAll with exists does the wrong thing, right? I assume that exists looks for any passing test, rather than requiring all generated values to pass. So I'd need to negate my test too in addition to adding exists, right? But I just tried that, and after printing a bunch of lines saying 0 tests passed, it finally reported test failure. Presumably that means it was unable to find an example where my property doesn't hold, right?

[CodaFi]

Ah, I remember why I didn't let you see this (even though SwiftCheck does keep track of it). The CheckerState struct was a mess for a while that was undocumented and in a crazy state of flux as I was adding and removing things to and from the framework. It's stabilized now, so it should be possible to implement something like QuickCheck's verboseCheckResult that will hand you back the last Result that happened rather than just pass it along SwiftCheck's internals.

I found the exists function, but presumably just replacing forAll with exists does the wrong thing, right?

Yeah. #43 Shows the identities for that stuff (if you'll excuse the notation).

Presumably that means it was unable to find an example where my property doesn't hold, right?

If that were the case, your log should contain "Could not satisfy existential"

[lilyball]

If that were the case, your log should contain "Could not satisfy existential"

Yeah it did:

Could not satisfy existential (after 500 tests):
<# some random arbitrary value description, I'm guessing the last one it tried #>

This was after printing 500 copies of

*** Passed 0 tests
.

[CodaFi]

Interesting. Seems I wasn't treating them like discards after all.

[CodaFi]

OK, so this still leaves you with the problem of how to reproduce a test case. I feel it's particularly inelegant to ask you to manually invoke some future verboseCheckResult. Perhaps a hatchway into the testing state could be used. Maybe some way of extending Property.withCallback? Do you have an elegant way you envision going about this?

[lilyball]

I feel like the most important thing I need is just the info necessary to deterministically re-run the tests (which presumably is just the initial seed and size; I haven't checked but I hope the RNG you're using is deterministic and local to the library, so if there's other threads running they can't interfere).

Once I have the seed and size, I can look into trying to replay it with a checker argument. But without knowing the seed/size, there's no way to replay. And if the failure is rare, then I can't just re-run it and hope to see the problem again.

I'm thinking this just needs to print the seed/size on a failure. No need to print anything different when things pass. After all, there's no benefit in replaying a successful check.

[CodaFi]

which presumably is just the initial seed and size; I haven't checked but I hope the RNG you're using is deterministic and local to the library, so if there's other threads running they can't interfere.

Lucky for you, it is neither of those things. At the time the little Random module in here was written, I was feeling neither confident in my ability to, nor ready to present, an implementation of a Mersenne twister or somesuch other feat of entropy-generating prowess. Nor was I willing to link with OpenSSL to accomplish the same. All this to say we use rand(). We should be using rand_r(). If you'd like to take a crack at it, see Random.swift.

After that is done, then the seed and RNG can be handed back to you in a useful manner.

[CodaFi]

The thing about printing them to the console is that you need both seed and StdGen to replay, and the latter is a runtime entity only. It makes sense to only print the seed, but I'm not sure what you'll gain by having the console spit it out.

[CodaFi]

@kballard With the merge of #120, you can now see seeds used for failing tests with the .verbose property, and you can initialize our new deterministic pseudo-RNG StdGen with those seed values to replay tests.

Thanks for the bug report!