Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Customizing Fuzzing Parameters via rv-config.json #7

Closed
moodmosaic opened this issue Aug 9, 2024 · 3 comments
Closed

Comments

@moodmosaic
Copy link
Member

To customize the rendezvous process, a new feature should be implemented that allows users to create an rv-config.json file in their project root. This file can be used to override default fuzzing parameters for specific contracts, such as setting custom min/max values for numbers.

Feature Details:

The rv-config.json file should support the following customization options:

  • Invariants: Specify which invariants to check during fuzzing.
  • Number: Set custom min and max values for number-based parameters.
  • String: Define minLength and maxLength for string-based parameters.
  • Seed: Set a custom seed for the random number generator to ensure reproducibility.
  • Iterations: Specify the number of iterations to run for each fuzzing session.
  • MaxBlocks: Limit the maximum number of blocks to be processed.
  • Timeout: Set a timeout duration for each fuzzing session.

Example rv-config.json:

{
  "customizations": {
    "contracts/foo.clar": {
      "invariants": ["invariant1", "invariant2"],
      "number": {
        "min": 0,
        "max": 100
      },
      "string": {
        "minLength": 5,
        "maxLength": 20
      },
      "seed": 12345,
      "iterations": 1000,
      "maxBlocks": 500,
      "timeout": 60
    },
    "contracts/bar.clar": {
      "invariants": ["invariant3", "invariant4"],
      "number": {
        "min": 10,
        "max": 50
      },
      "string": {
        "minLength": 3,
        "maxLength": 15
      },
      "seed": 54321,
      "iterations": 2000,
      "maxBlocks": 1000,
      "timeout": 120
    }
  }
}

Benefits:

This feature will allow running rendezvous in parallel using different profiles, making it potentially faster to uncover edge cases and increasing the efficiency of the fuzzing process.

Acceptance Criteria:

  • The rv-config.json file should be recognized and parsed correctly when placed in the project root.
  • Custom parameters specified in rv-config.json should override the default fuzzing settings.
  • Ensure that the tool can handle multiple profiles and run them in parallel if specified.
@BowTiedRadone
Copy link
Collaborator

When we move forward with implementing this feature, we can introduce an option for users to specify eligible trait definitions. Currently, this is the only Clarity type not automatically generated by our generators. By allowing users to specify Clarity traits in the JSON configuration file, we can select from these specified traits when calling functions that require trait definitions.

@moodmosaic
Copy link
Member Author

I’d rather close this and split it into two issues:

  • Support for traits
  • Support for user-defined generators

I lean towards skipping rv-config.json entirely, as it’s limited to basic overrides for numbers and strings.

Instead, the user can specify a path (e.g., via --vars or --gens for custom generators). Custom generators replace the built-in ones (if they exist). If no built-in generator is present, the custom generator simply adds to the fast-check loop.

In this case, support for traits means the user writes a custom generator.

Combined with #60, achieves the same result as rv-config.json but in a more explicit way.


One issue with rv-config.json is that its complexity can quickly grow. As an old colleague once said: "It’s like adding more knobs to a machine—too many, and it all falls apart." (See Mike Hadlow’s Configuration Complexity Clock for more.)

@moodmosaic
Copy link
Member Author

Closing as per #66, which also supports #65. We can reopen if that direction isn't what we actually want. A config file can always be added later if really needed, though it's harder to remove or modify once it's in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants