Add `test_without_fail_case` lint to check if a test actually has a way to fail or not #13579

kayagokalp · 2024-10-21T23:57:22Z

related to #12484.

changelog: Adds a test_without_fail_case lint which checks if a test can fail or not. If failure is not possible, the test is linted. This is done to prevent any silently failing tests that actually was not asserting anything, improving test suite quality.

rustbot · 2024-10-21T23:57:26Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @blyxyas (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

README.md

kayagokalp · 2024-10-24T21:44:16Z

@rustbot author

kayagokalp · 2024-10-24T21:44:40Z

I am adding configuration capabilities to this PR and will also add more tests.

blyxyas

Good starting point, but I'm worried for performance and memory usage. I'm trying to think of a way to check if a function links to the panic handler DefId.

blyxyas · 2024-10-25T09:12:34Z

book/src/lint_configuration.md

+Whether to consider indexing as a fallible operation while assesing if a test can fail.
+Indexing is fallible, and thus the a test that is doing that can fail but it is likely
+that tests that fail this way were not intended.
+
+If set true, the lint will consider indexing into a slice a failable case
+and won't lint tests that has some sort of indexing. This analysis still done
+in a interprocedural manner. Meaning that any indexing opeartion done inside of
+a function that the test calls will still result the test getting marked fallible.
+
+By default this is set to `false`. That is because from a usability perspective,
+indexing an array is not the intended way to fail a test. So setting this `true`
+reduces false positives but makes the analysis more focused on possible byproducts
+of a test. That is the set of operations to get the point we assert something rather
+than the existance of asserting that thing.


I think it would be excellent to have a way to, instead of adding a third configuration option, be able to use std::ops::Index::index or similar in the other two configuration options.

Anyway, this is way too complex.

Suggested change

Whether to consider indexing as a fallible operation while assesing if a test can fail.

Indexing is fallible, and thus the a test that is doing that can fail but it is likely

that tests that fail this way were not intended.

If set true, the lint will consider indexing into a slice a failable case

and won't lint tests that has some sort of indexing. This analysis still done

in a interprocedural manner. Meaning that any indexing opeartion done inside of

a function that the test calls will still result the test getting marked fallible.

By default this is set to `false`. That is because from a usability perspective,

indexing an array is not the intended way to fail a test. So setting this `true`

reduces false positives but makes the analysis more focused on possible byproducts

of a test. That is the set of operations to get the point we assert something rather

than the existance of asserting that thing.

Whether to consider indexing (`a[b]`) as a fallible operation while checking if a test can fail.

Indexing is fallible, and thus it can panic, but this panic is likely not intended to be tested.

I think it would be excellent to have a way to, instead of adding a third configuration option, be able to use std::ops::Index::index or similar in the other two configuration options.

I agree, I'll think about it

book/src/lint_configuration.md

clippy_lints/src/test_without_fail_case.rs

blyxyas · 2024-10-25T09:37:28Z

clippy_lints/src/test_without_fail_case.rs

+                indexing_fallible: conf.test_without_fail_case_include_indexing_as_fallible,
+                fallible_paths: conf.test_without_fail_case_fallible_paths.iter().cloned().collect(),
+                non_fallible_paths: conf.test_without_fail_case_non_fallible_paths.iter().cloned().collect(),


Is there a reason that these are FxHashSet?

Mostly out of habit of using FxHashSet in the compiler for performance reasons. But given that this is only the configuration I'll change it into HashSet.

I meant: Is there a reason is it not a simple Vec? We prefer FxHashSet to HashSet, but I just see no use for this to be a hashed collection. (Unless there's a very performance improvement I'm not aware of)

I guess it is a small lookup table, as the list coming from configuration file likely to be limited, can't see a reason why it couldn't be a Vec for sure. Can change it.

blyxyas · 2024-10-25T10:28:22Z

clippy_lints/src/test_without_fail_case.rs

+    type NestedFilter = nested_filter::OnlyBodies;
+
+    fn visit_expr(&mut self, expr: &'tcx Expr<'_>) {
+        if self.fail_found {


Could you use ControlFlow<()> instead of self.fail_found?

The Visitor trait accepts it :)

blyxyas · 2024-10-25T10:29:10Z

clippy_lints/src/test_without_fail_case.rs

+
+impl<'tcx> Visitor<'tcx> for SearchFailIntraFunction<'_, 'tcx> {
+    type NestedFilter = nested_filter::OnlyBodies;
+


Suggested change

type Result = ControlFlow<()>;

clippy_lints/src/test_without_fail_case.rs

Co-authored-by: Alejandra González <[email protected]>

kayagokalp · 2024-10-28T10:53:17Z

Good starting point, but I'm worried for performance and memory usage. I'm trying to think of a way to check if a function links to the panic handler DefId.

If there is way to see if a function links to a panic handler without visiting them explicitly that would be great! Otherwise to access the performance characteristics it might be wise to take a look at set of benchmarks. I am curious to see how many interleaved levels of functions calls are average/likely to be upper limit.

flip1995 · 2024-10-28T10:51:40Z

CHANGELOG.md

+[`test-without-fail-case-fallible-paths`]: https://doc.rust-lang.org/clippy/lint_configuration.html#test-without-fail-case-fallible-paths
+[`test-without-fail-case-include-indexing-as-fallible`]: https://doc.rust-lang.org/clippy/lint_configuration.html#test-without-fail-case-include-indexing-as-fallible
+[`test-without-fail-case-non-fallible-paths`]: https://doc.rust-lang.org/clippy/lint_configuration.html#test-without-fail-case-non-fallible-paths


I find 3 configuration values for this lint a bit much. Also those are a mouth full.

I think there should only one configuration option at most: A list of items/functions that should prevent triggering the lint.

flip1995 · 2024-10-28T10:54:05Z

tests/ui/test_without_fail_case.rs

+fn implicit_panic() {
+    panic!("this is an implicit panic");
+}


I would expect that most of those functions, that move the assertion or panic out of the #[test] function is defined inside the same module as the tested code. Pruning the functions this lint checks with this heuristic, whether they panic, might prevent big performance impacts this lint might have.

flip1995 · 2024-10-28T10:55:45Z

clippy_lints/src/test_without_fail_case.rs

+    fn check_item(&mut self, cx: &LateContext<'tcx>, item: &'tcx Item<'tcx>) {
+        // Only interested in functions that are annotated with `#[test]`.
+        if let ItemKind::Fn(_, _, body_id) = item.kind
+            && is_in_test_function(cx.tcx, item.hir_id())


This lint only runs on tests. This also means, that the lint can only detect something, if Clippy is run with cargo clippy --all-targets or cargo clippy --tests. This should be mentioned in the documentation of the lint.

I'm not sure, if lintcheck is run with --all-targets. It at least didn't trigger once for all the crates we test on.

rustbot assigned blyxyas Oct 21, 2024

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties label Oct 21, 2024

kayagokalp changed the title ~~Kayagokalp/test without panic~~ Add test_without_fail_case lint to check if a test actually has a way to fail or not Oct 21, 2024

kayagokalp mentioned this pull request Oct 22, 2024

Check for possible silently passing invalid tests by making sure some sort of panic! exists in tests #12484

Open

kayagokalp commented Oct 22, 2024

View reviewed changes

README.md Show resolved Hide resolved

kayagokalp marked this pull request as ready for review October 22, 2024 00:01

kayagokalp marked this pull request as draft October 22, 2024 00:02

kayagokalp marked this pull request as ready for review October 22, 2024 00:16

rustbot added S-waiting-on-author Status: This is awaiting some action from the author. (Use `@rustbot ready` to update this status) and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties labels Oct 24, 2024

blyxyas requested changes Oct 25, 2024

View reviewed changes

feat: implement test_without_fail case lint

1ab9ae6

kayagokalp force-pushed the kayagokalp/test-without-panic branch from d2c1c4b to 1ab9ae6 Compare October 26, 2024 19:05

doc: apply doc related suggestions from code review

aa73111

Co-authored-by: Alejandra González <[email protected]>

flip1995 reviewed Oct 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `test_without_fail_case` lint to check if a test actually has a way to fail or not #13579

Add `test_without_fail_case` lint to check if a test actually has a way to fail or not #13579

kayagokalp commented Oct 21, 2024 •

edited

Loading

rustbot commented Oct 21, 2024

kayagokalp commented Oct 24, 2024

kayagokalp commented Oct 24, 2024

blyxyas left a comment

blyxyas Oct 25, 2024

kayagokalp Oct 26, 2024

blyxyas Oct 25, 2024

kayagokalp Oct 26, 2024

blyxyas Oct 28, 2024

kayagokalp Oct 28, 2024

blyxyas Oct 25, 2024

blyxyas Oct 25, 2024

kayagokalp commented Oct 28, 2024

flip1995 Oct 28, 2024

flip1995 Oct 28, 2024

flip1995 Oct 28, 2024


		impl<'tcx> Visitor<'tcx> for SearchFailIntraFunction<'_, 'tcx> {
		type NestedFilter = nested_filter::OnlyBodies;

Add test_without_fail_case lint to check if a test actually has a way to fail or not #13579

Are you sure you want to change the base?

Add test_without_fail_case lint to check if a test actually has a way to fail or not #13579

Conversation

kayagokalp commented Oct 21, 2024 • edited Loading

rustbot commented Oct 21, 2024

kayagokalp commented Oct 24, 2024

kayagokalp commented Oct 24, 2024

blyxyas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kayagokalp commented Oct 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add `test_without_fail_case` lint to check if a test actually has a way to fail or not #13579

Add `test_without_fail_case` lint to check if a test actually has a way to fail or not #13579

kayagokalp commented Oct 21, 2024 •

edited

Loading