feat: Introduce a way to suppress violations #119
Conversation
|
|
|
|
||
| Furthermore, we are adding one more reason to exit with an error code (see "Maintaining a lean baseline"). This might have some negative side-effects to wrapper scripts that assume that error messages are available when that happens. We could introduce a different exit code, to differentiate between exiting due to unresolved errors or ignored errors that do not occur anymore. | ||
|
|
||
| ## Alternatives |
There was a problem hiding this comment.
the alternative we've introduced at Canva is that we designate specific rules as "in migration" and we only consider reports from those rules if they exist in changed files (according to git comparison against the main branch).
With this system developers must address lint errors if they touch a file but otherwise they can be ignored.
This does require integration with the relevant source control system - though we've found it works quite well.
|
Thanks for the RFC @softius. Can you please sign the CLA as requested in this comment? |
|
@octogonz Would really like your feedback on this based on your work on Rush Stack Bulk. |
|
The feature that we designed is part of @rushstack/eslint-patch. Kevin Yang summarized it in this blog post. A key design question is how to keep bulk suppressions stable across Git merges, since they are stored separately from the associated source code. From a cursory glance, it sounds like this RFC is proposing to count the number of errors per file path. We prototyped that design, as well as a design that tracks line #'s, and also a design that tracks formatted messages ( Ultimately we settled on a JSON file using a There's probably still a lot that can be improved about our approach. However I can say that Microsoft and TikTok are using it successfully in two very large TypeScript monorepos, where style guide migrations involve hundreds of thousands of source files and megabytes of JSON. So it's worth studying even if you don't adopt it. 🙂 As far as future improvements, a feature I'd like to see is an API that enables the VS Code extension to highlight suppressed violations specially. Currently people either highlight them as normal errors (which I find confusing, since you have to run the CLI to find out whether you really need to fix it or not) or else don't highlight them at all. |
|
@octogonz thanks for the insights. I think we're past the point of adopting eslint-bulk wholesale, but would welcome your feedback on this RFC to see if there are any salient points that have been missed or problem areas you've encountered. |
Agreed, the I've been really busy lately, but I'd like to help out with the RFC. Let me make some time to read through @softius's doc in more detail, then follow up with feedback. |
|
@softius just a reminder that there are some suggestions for you to apply. Please, let us know if you need any help. |
|
I really support this RFC - we've been running our own baseline package (just a wrapper around eslint), which takes the same approach. We've been using this implementation on about 30+ prod projects with no complaints from our team. Hopefully, this implementation can assist with the RFC |
Co-authored-by: Nicholas C. Zakas <[email protected]>
There was a problem hiding this comment.
Apologies for the delay in getting back to this. Had a lot of post-v9 work.
I think there are some helpful ideas that eslint-bulk has that we should consider here:
- Using the term "suppressions" instead of "baseline" - I think this clarifies what is actually happening. While people may not understand what generating a baseline is, most will understand when something is suppressed.
- A way to indicate which rules should be suppressed. For instance, if I just want to suppress "no-eval", I should be able to do that. That may mean rethinking the command line flags. So maybe
--suppress-rule <ruleId>and--suppress-allfor all rules?
|
@Humni that's great! Do you have any specific feedback on the implementation described in this proposal? How does it compare to what you're already using? |
|
Hi Nicholas, thanks for checking out the eslint-bulk project!
Other names we considered were exemptions, pardons, and legacy-suppressions. On a semi-related note, the "baselines" feature seems to very closely resemble the already existing eslint --max-warnings flag, which might be a source of confusion if the two features are both retained. |
@kevin-y-ang I'm not sure I follow. Can you explain what you mean by this? |
|
At a conceptual level, both the proposed baseline feature and the --max-warnings feature set a threshold of X errors where the system will alarm if there are >X errors. At our company, we previously used The message essentially being conveyed here is "Okay there are already X number of ESLint warnings in this package but we won't allow any more", while the message being conveyed with the baseline feature seems to be a more granular version of this: "Okay there are already X number of ESLint errors/warnings for rule R in file F but we won't allow any more." Anyway, it's just a similarity I see, it's not a big deal. |
Probably the main decision to make for this RFC which style of error matching you use. These would be:
With our current implementation, we use the Exact Error Line Matching, which does have some short falls. It works pretty well, the vast majority of PRs don't require any rework. There are cases where if you modify a line at the top of a file though, will cause all of the existing errors to be exposed. This approach was chosen due to simplicity. For the most part this does push a project to having a smaller and smaller baseline, however it does lead to developers touching the baseline "too much" due to them not wanting to fix errors not directly related to their PR. We're currently exploring moving to a context aware error matching, so it only comes up if the nearest 3 lines of code are modified - LuminateOne/eslint-baseline#8. This would provide a solution to the issue above, but it's not yet tested but seems like a sensible approach (if it can be done in a performant way). I would highly recommend an error count approach the same as PHPStan (as per comments from @ondrejmirtes), as that seems to sit with the development team better. Only other question "feature request" I would add into this is to allow for "flushing" a baseline of excess/resolved errors only (and not add any new ones). Just a QOL improvement though as it's typically easy enough to see new lines added into the diff in the baseline. |
|
@Humni gotcha, that's helpful. So it sounds like the current RFC, which uses error counting, is what you'd prefer. 👍 Keep in mind, too, that we generally build things incrementally. Our goal here is to get a baseline (no pun intended) of support for suppressions such that we can continue to build on top of it as feedback comes in. |
|
@softius are you still working on this? There is some outstanding feedback to review. |
|
@eslint/eslint-tsc looking for another approval on this. |
|
|
||
| ### Suppressing violations of a specific rule | ||
|
|
||
| A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. |
There was a problem hiding this comment.
| A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. | |
| A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this option can accept an array of string values. |
| * The developer runs `eslint --supress-all ./src` to create the suppressions file. | ||
| * Running `eslint ./src` reports no violations and exits with status 0. | ||
| * After fixing a violation, the suppressions file still contains the now-resolved violation. | ||
| * Running `eslint ./src` again reports no violations but exits with a non-zero status code, indicating the suppressions file needs updating. |
There was a problem hiding this comment.
In this case, will there be an error message, and what would it look like? Technically, will it be a lint message passed to the formatter along with other lint messages for the file, or a separate output?
There was a problem hiding this comment.
In elm-review, if counts go down, then this file is automatically updated when running the process. The reason I chose to do so is because I don't want to annoy the users when they do the right thing.
Say I fix an issue, I run eslint (which on a large project could take a significant amount of time), then I get to hear that I did a good thing by solving problems, but that I need to do the process again. I believe this can be frustrating and I wanted to avoid that.
So we update the file automatically. The downside is that the file might not get checked in properly. To help with that, we recommend that people in CI (or in their test suite) run elm-review suppress --check-after-tests (after running elm-review) which is quick because it only invokes git status --short -- review/suppressed/ and reports an error if suppression files should have been committed. (Surprisingly, so far no-one has asked for supporting other VCSs).
Also, we try to display a friendly message when you fixed an issue: "There are still 316 suppressed errors to address, and you just fixed 1!" (I could have added emojis but I also didn't want to go over the top).
That's a good question. I think it makes sense to insert lint messages suppressed by this feature into |
I agree. Makes sense. 👍 |
Makes sense. I think that errors suppressed by the suppressions file will have no |
|
@softius I think all that's left is to update the RFC to mention |
|
@softius Any way we can help you to finish the work on this RFC? |
Co-authored-by: Milos Djermanovic <[email protected]>
|
RFC updated to indicate that matched messages will be moved from |
There was a problem hiding this comment.
Overall LGTM. Would like @mdjermanovic and @fasttime to verify before merging.
Thanks so much for all of your work and patience.
|
I think the following review comments haven't been addressed: |
| 2. **Match Errors Against Suppressions** | ||
| * For each error, check if the error and the file are in the suppressions file. | ||
| * If yes, decrease count, in memory, by 1 and move the message to `LintResult#suppressedMessages` unless count is zero. | ||
| * If no, keep the error. |
There was a problem hiding this comment.
(related to the discussion under #119 (comment))
For one file and one rule, I think either all or none of the messages should be moved to suppressedMessages. By this algorithm, when the number of error messages is greater than the number stored in the suppressions file, some would be moved to suppressedMessages while some would stay in messages.
|
|
||
| The suggested solution always compares against the existing suppressions file, typically `.eslint-suppressions.json`, unless `--suppressions-location` is specified. This makes it easier for existing and new projects to adopt this feature without the need to adjust scripts in `package.json` and CI/CD workflows. | ||
|
|
||
| To perform the comparison, we will go through each result and message from `ESLint.lintFiles`, checking each error `(severity == 2)` against the suppressions file. By design, we ignore warnings since they don't cause eslint to exit with an error code and serve a different purpose. If the file and rule are listed in the suppressions file, we can move the message to `LintResult#suppressedMessages` and ignore the result message. |
There was a problem hiding this comment.
SuppressedLintMessage should have an additional property suppressions. We should specify what value it will have in this case. I think it can be:
suppressions: [{ kind: "file", justification: "" }]There was a problem hiding this comment.
@mdjermanovic can you please clarify what would be the purpose of this new property?
There was a problem hiding this comment.
It describes the reason why the lint message is suppressed. It's a mandatory property in SuppressedLintMessage type (objects in LintResult#suppressedMessages). Without it, integrations and formatters that use LintResult#suppressedMessages could break.
There was a problem hiding this comment.
Thanks @mdjermanovic . I was under the impression that you were referring to a new property for some reason. That is clear now.
|
@mdjermanovic as it seems like we are getting down to fine-tuning some details, can we move this to Final Commenting? |
There was a problem hiding this comment.
I'm fine with moving to Final Commenting with a note that when the number of errors exceeds the allowed number, all errors should be reported, as described in #119 (comment).
| * If `--prune-suppressions` is passed, take the updated suppressions from memory to check which suppressions are left. | ||
| * For each suppression left, update the suppressions file by either reducing the count or removing the suppression. | ||
| 4. **Report and exit** | ||
| * Exit with a non-zero status if there are unmatched suppressions, optionally listing them in verbose mode. |
There was a problem hiding this comment.
By verbose mode we probably mean when the --debug flag is passed on the command line.
| /** | ||
| * Removes old suppressions that do not occur anymore. | ||
| * @returns {void} | ||
| */ | ||
| prune() |
There was a problem hiding this comment.
I think this method needs to accept an argument with the updated count of suppressed errors per rule and per file.
| Here is a high-level overview of the execution flow: | ||
|
|
||
| 1. **Check for Options** | ||
| * If both `--suppress-all` and `--suppress-rule` are passed, exit with an error (these options are mutually exclusive). |
There was a problem hiding this comment.
It seems that --prune-suppressions will have no effect when --suppress-all or --suppress-rule is passed. Should it be also disallowed when one of the other options is passed?
rfcs/designs/2024-baseline-support/README.md
Lines 224 to 230 in f134daa
| log.error("There are left suppressions that do not occur anymore. Consider re-running the command with `--prune-suppressions`."); | ||
| return 2; |
There was a problem hiding this comment.
Question: do we really want to exit without printing the lint results here?
|
The final commenting period has completed, so I'm merging this. The remaining questions are implementation issues that we can discuss on the actual implementation. @softius thanks for your patience. Please feel free to submit an implementation PR when ready. |
Summary
Suppress existing violations, so that they are not being reported in subsequent runs. It allows developers to enable one or more lint rules and be notified only when new violations show up.
Related Issues