Collapse output to 20 items #160

daniilsapa · 2025-01-02T23:23:43Z

Closes #51

The solution is still raw and it's an early PR, so you can see the general approach and we can clarify some details.

When I was working on this logic I came across a thought that we probably need to make the number we collapse diagnostics to configurable and introduce a new config object type for such things that configure Steiger itself (it is very likely that other options will appear pretty soon) and not severities/options for rules. It seems it's better to introduce such a new object type earlier, e.g. in the next minor update.
Should we introduce severity to consideration when we collapse diagnostics? Looks like we need to count error first because it's the thing that breaks CIs or local checks. It would be quite annoying if you do not see an error because the quota is exceeded with warnings
It looks like runRules from app.ts needs to return some Report interface, so it contains not just diagnostics but also some meta info about them. We need this to display the number of shown vs hidden diagnostics etc. other fields may appear in the future.

changeset-bot · 2025-01-02T23:23:46Z

🦋 Changeset detected

Latest commit: 8d3d231

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@steiger/pretty-reporter	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

illright · 2025-01-03T14:02:38Z

Interesting considerations, thanks for the early update! I looked at the code and realized that I forgot to mention one important detail in the issue description.

I originally envisioned this as a purely cosmetic thing rather than a linter feature, and as such, I planned that this functionality would be added to the pretty reporter rather than the linter core. This way, all reporters get the same full diagnostics, but some reporters might choose to hide some.

The inspiration for this feature came from Biome, they also have a 20-diagnostic limit. Here's how they show it:

That's why I don't think we should make this thing configurable or even disable-able. If one wants to see all diagnostics, I think it's not because they want to read all of them individually. For mass-reading of diagnostics, the JSON reporter is probably better. I suggest we write "X diagnostics were hidden, run with --reporter json to see all diagnostics" or something.

The severity consideration is a very good point, I agree that we probably want to sort the errors first and then, if there's space left, print warnings too. If there's a rule that has produced only warnings, then we only print its diagnostics if all the errors have been printed.

daniilsapa · 2025-01-03T15:09:17Z

Oh, yeah, actually it makes sense to make it a cosmetic thing and move pretty-reporter. It would also make the implementation easier and the resulting codebase less complex. Do you like the general approach? If so, I will move the logic to pretty-reporter, add severity to consideration, and refactor it a bit.

illright · 2025-01-03T19:43:12Z

Overall the code looks good, I only have minor concerns about over-abstracting — such as having an entire feature for this, and also making the quota a modifiable parameter. Seems like the code could be simpler by keeping it all in one function

…e solution

daniilsapa · 2025-01-06T14:53:08Z

@illright
I've moved the solution to pretty-reporter. Also, you can see the output here:

Let me know if you think we must rephrase/change it somehow

daniilsapa · 2025-01-06T14:55:11Z

The bundle size pipeline fails when the git branch contains slashes. I'll fix the problem as part of this issue

illright

Thanks for noticing the bundle size workflow issue, my mistake! I think we can replace everything that isn't /[0-9a-zA-Z-._]/ (ref) with - to overcome this.

The solution is good overall, but slightly hard to understand, I made some suggestions to improve clarity, let me know what you think

illright · 2025-01-06T17:22:05Z

packages/pretty-reporter/src/index.ts

+  const autofixable = collapsedDiagnostics.filter((d) => (d.fixes?.length ?? 0) > 0)
+  if (autofixable.length === collapsedDiagnostics.length) {


issue: if I understand correctly, this code will only report how many of the printed 20 diagnostics are fixable. This might be misleading because the autofix engine doesn't know about this collapsing, so it will apply more fixes than what's promised by the output

suggestion: I think we should count and apply fixes from all diagnostics, not just the collapsed ones

illright · 2025-01-06T17:26:10Z

packages/pretty-reporter/src/index.ts

-    '\n'
+    '\n ' +
+    (collapsedDiagnosticsCount < initialDiagnosticsCount
+      ? `${pc.reset(initialDiagnosticsCount - collapsedDiagnosticsCount)} diagnostics are not shown in the report as they exceed the limit allowed by Steiger`


suggestion: yeah, I had an idea to actually tell people how to see all diagnostics instead of this technical "exceed the limit" stuff that Biome prints

Suggested change

? `${pc.reset(initialDiagnosticsCount - collapsedDiagnosticsCount)} diagnostics are not shown in the report as they exceed the limit allowed by Steiger`

? `${pc.reset(initialDiagnosticsCount - collapsedDiagnosticsCount)} diagnostics are not shown, use ${pc.bold(pc.green('--reporter json'))} to see them.`

issue: this will print "1 diagnostics" when only 1 diagnostic was hidden

suggestion: use the s function (imported in this file) to conditionally pluralize the word

illright · 2025-01-06T20:52:06Z

packages/pretty-reporter/src/collapse-diagnostics.ts

+  const quotaPerBucket = buckets.slice(0).fill(0)
+  let remainingQuota = quota
+
+  for (let i = 0; remainingQuota > 0; i += 1) {


issue: the fact that this is a for loop has made it quite difficult for me to understand the rest of the loop body. A for loop is meant to take a counter variable from A to B, but this one only specifies the A, so it's not a for loop in the traditional sense. Other strange things arise as a consequence, such as assigning i = -1, which is not a valid index, but will be made valid by the mechanics of the for loop

suggestion: let's make this a while loop, I think it more accurately reflects what we're trying to do here:

while (there is quota left) { assign one unit of quota }

illright · 2025-01-06T20:53:19Z

packages/pretty-reporter/src/collapse-diagnostics.ts

+    // If it ran out of the items earlier than the quota, break
+    if (allItems === quota - remainingQuota) {
+      break
+    }


issue: this condition also broke my head for a solid couple of minutes :D

suggestion: instead of checking this on every iteration, let's instead assign remainingQuota = Math.min(quota, allItems), such that there is never more quota than the total number of items

illright · 2025-01-06T20:57:52Z

packages/pretty-reporter/src/collapse-diagnostics.ts

+  return diagnostics.slice(0).sort((a, b) => severityOrder[a.severity] - severityOrder[b.severity])
+}
+
+// Export for testing purposes to be able to pass a custom quota


issue: I'm not a big fan of using one thing but testing another, it slightly undermines my confidence in the tests

suggestion: how about we make collapseDiagnostics accept the quota as an argument, and move the 20 constant to src/index.ts instead?

illright · 2025-01-06T21:01:42Z

packages/pretty-reporter/src/collapse-diagnostics.test.ts

+
+const defaultSeverity = 'error' as 'warn' | 'error'
+
+// dummy rules


suggestion (non-blocking): how about we instead write a function to generate dummy diagnostics with a simple index in the message and then call it in the test setup? This way we can even test with quota = 20

function makeDiagnostic(ruleName: string, index: number): PartialDiagnostic

const diagnosticPerRule = [ new Array(15).fill(0).map((_, index) => makeDiagnostic('rule-1', index)), new Array(10).fill(0).map((_, index) => makeDiagnostic('rule-2', index)), ]

And in the tests that actually drop some diagnostics, we can assert by doing diagnosticPerRule[0].slice(0, 5). This way we also test that it doesn't change the order of diagnostics within one rule

illright · 2025-01-06T21:05:03Z

packages/pretty-reporter/src/collapse-diagnostics.test.ts

+
+    const result = trimDiagnosticsToMeetQuota(diagnosticPerRule, 3)
+
+    expect(result).toEqual([[rule1Message1], [rule2Message1], [rule3Message1]])


note: for some reason, this test fails for me locally because result also includes an empty array at the end

Implement diagnostic quota logic

e55b9fa

daniilsapa requested a review from illright January 2, 2025 23:23

daniilsapa added 7 commits January 4, 2025 17:50

Move diagnostic collapsing logic to pretty-printer

f3bdce4

Add severity to consideration when collapsing the report, refactor th…

9d837bd

…e solution

Add a changeset

43e7f83

Merge branch 'next' into feature/collapse-output

f22685c

Fix conflicts in pnpm-lock.yaml

75db8ef

Fix vitest version for pretty-reporter

66bbf39

Fix pnpm-lock issue

8d3d231

illright requested changes Jan 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collapse output to 20 items #160

Collapse output to 20 items #160

daniilsapa commented Jan 2, 2025 •

edited

Loading

changeset-bot bot commented Jan 2, 2025 •

edited

Loading

illright commented Jan 3, 2025

daniilsapa commented Jan 3, 2025

illright commented Jan 3, 2025

daniilsapa commented Jan 6, 2025

daniilsapa commented Jan 6, 2025

illright left a comment

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

illright Jan 6, 2025

		const autofixable = collapsedDiagnostics.filter((d) => (d.fixes?.length ?? 0) > 0)
		if (autofixable.length === collapsedDiagnostics.length) {

	? `${pc.reset(initialDiagnosticsCount - collapsedDiagnosticsCount)} diagnostics are not shown in the report as they exceed the limit allowed by Steiger`
	? `${pc.reset(initialDiagnosticsCount - collapsedDiagnosticsCount)} diagnostics are not shown, use ${pc.bold(pc.green('--reporter json'))} to see them.`


		const defaultSeverity = 'error' as 'warn' \| 'error'

		// dummy rules


		const result = trimDiagnosticsToMeetQuota(diagnosticPerRule, 3)

		expect(result).toEqual([[rule1Message1], [rule2Message1], [rule3Message1]])

Collapse output to 20 items #160

Are you sure you want to change the base?

Collapse output to 20 items #160

Conversation

daniilsapa commented Jan 2, 2025 • edited Loading

changeset-bot bot commented Jan 2, 2025 • edited Loading

🦋 Changeset detected

illright commented Jan 3, 2025

daniilsapa commented Jan 3, 2025

illright commented Jan 3, 2025

daniilsapa commented Jan 6, 2025

daniilsapa commented Jan 6, 2025

illright left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniilsapa commented Jan 2, 2025 •

edited

Loading

changeset-bot bot commented Jan 2, 2025 •

edited

Loading