Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add discard functionality for conditional tests #40

Merged
merged 21 commits into from
Nov 8, 2024

Conversation

BowTiedRadone
Copy link
Collaborator

@BowTiedRadone BowTiedRadone commented Nov 1, 2024

This PR adds logic to discard property tests that don't meet certain preconditions. It partially addresses #38.

Sample output

When a test is discarded, it is now highlighted with a [WARN] and dimmed function arguments:

Screenshot 2024-11-04 at 6 06 17 PM

This commit introduces an example of a Clarity preliminary function designed
to pair with a conditional property test. It also includes a comment outlining
the rules for structuring preliminary functions, covering their requirements.
This commit updates the conditional property test function by replacing its
early return condition with a call to the corresponding preliminary function.
This change clarifies the role of preliminary functions in validating input
before executing the main test logic.
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
property.ts Show resolved Hide resolved
This commit adds checks for preliminary functions to ensure they follow the
rules. It checks that the parameters match those of the test functions and
that the return type is a boolean.
This commit adds the preliminary function run before the test function and
stores the result.
property.ts Outdated Show resolved Hide resolved
@moodmosaic
Copy link
Member

moodmosaic commented Nov 1, 2024

Made some optional/stylish tweaks here. Can be tested with:

npm install && ./example/watchdog.sh ./rv example/ slice --type=test

image


Here is an alternative one:

image

@moodmosaic
Copy link
Member

Perhaps dim the generated args for discarded tests and that’s it. No need to even print (discarded) at all.

BowTiedRadone and others added 4 commits November 2, 2024 13:41
This commit adds conditional logic to property testing based on preliminary
functions' response. Property tests with a preliminary function won’t need an
initial condition. The updated `test-slice-list-int` demonstrates this new
setup. This commit also includes a minor indentation fix.
This commit updates the `ansicolors` used for property testing logging.

Thanks to @moodmosaic for this great proposal:
3adcfe4.
@BowTiedRadone BowTiedRadone marked this pull request as ready for review November 4, 2024 16:02
@BowTiedRadone BowTiedRadone changed the title [DRAFT] Add discard functionality for conditional tests Add discard functionality for conditional tests Nov 4, 2024
Copy link
Member

@moodmosaic moodmosaic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a high-level review. Have to decide on the DSL (is it can-, discard-, preliminary?). Can't mix and match those terms as they are core terminology for RV.

example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
example/contracts/slice.tests.clar Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.ts Show resolved Hide resolved
property.ts Outdated Show resolved Hide resolved
property.tests.ts Outdated Show resolved Hide resolved
This commit adds tests for `isParamsMatch` and `isReturnTypeBoolean`.
If there are discarded tests, output is green and yellow but the result
still shows as green. Better to leave it uncolored.
Copy link
Member

@moodmosaic moodmosaic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Massive! Final touches added and ready to go.


Excited to use this for sBTC invariant/property-based fuzzing, expanding on
what we achieved with PoX-4 fuzzing techniques.

property.tests.ts Show resolved Hide resolved
property.ts Show resolved Hide resolved
@moodmosaic moodmosaic merged commit 24cbd52 into master Nov 8, 2024
@moodmosaic moodmosaic deleted the feat/discard-tests branch November 8, 2024 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants