In Rider or ReSharper theory where all test cases were skipped is displayed as failed because SkipException increases TestsFailed counter

[ivanskorikov]

When using Rider or R# test runner, a theory with skipped test cases is displayed as containing failing tests.

public class Test
{
	[SkippableTheory]
	[MemberData(nameof(WorkingData))]
	public void WorkingTest(IEnumerable<string> a)
	{
		throw new SkipException();
	}

	[SkippableTheory]
	[MemberData(nameof(FailingData))]
	public void FailingTest(IEnumerable<string> a)
	{
		throw new SkipException();
	}

	public static TheoryData<IEnumerable<string>> WorkingData()
	{
		return new() { Array.Empty<string>() };
	}

	public static TheoryData<IEnumerable<string>> FailingData()
	{
		return new() { Enumerable.Empty<string>() };
	}
}

The TestCaseFinished event gets fired two times: once for FailingTest and once for WorkingTest.

For FailingTest, the display property is set FailingTest, which causes it to match to the parent node. And non-zero TestsFailed is why we interpret it as failed.

For WorkingTest, display is WorkingTest(a: []). And although TestsFailed is also non-zero, it's the second time a result gets reported for that element (the first being the dedicated OnTestSkippedEvent), so we discard it entirely.

Please consider handling SkipException in a way that would not count towards TestsFailed counter.

Originally reported here: RIDER-99523 Skipped theory tests are marked as failed when an IEnumerable is used as a value (Xunit.SkippableFact)

[AArnott]

Thanks for this thorough report. I'm surprised to learn that these tests are reported as failed. I can look into that. I'm not sure when I'll get to it. I'd welcome a PR.

[AArnott]

This is how SkippableFact avoids this problem:

Xunit.SkippableFact/src/Xunit.SkippableFact/Sdk/SkippableFactTestCase.cs

Lines 77 to 78 in 0a2cdce

	result.Failed -= messageBusInterceptor.SkippedCount;
	result.Skipped += messageBusInterceptor.SkippedCount;

SkippableTheory has exactly the same code:

Xunit.SkippableFact/src/Xunit.SkippableFact/Sdk/SkippableTheoryTestCase.cs

Lines 75 to 76 in 0a2cdce

	result.Failed -= messageBusInterceptor.SkippedCount;
	result.Skipped += messageBusInterceptor.SkippedCount;

But in the latter case, this code is not called. I don't yet know why. But at least that partly explains why they are reported as failures in Rider.

[AArnott]

Ok, for reasons unknown, although SkippableTheoryTestCase isn't called, the SkippableFactTestCase is called. So either way, the code to decrement Failed and increment Skipped executes.
So I don't know what might be going wrong in R# or Rider. I don't have a license to either so I can't investigate further.

If Jetbrains were to comp me a license, I'd be happy to investigate further.

[0xced]

@ivanskorikov Is this still an issue with Rider 2024.3 (Build #RD-243.21565.191)? I could not reproduce your issue.

Also @AArnott, JetBrains announced that WebStorm and Rider Are Now Free for Non-Commercial Use last month.

[dArignac]

@0xced I have the same issue and I am still seeing this in Rider 2024.3 (Build #RD-243.21565.191).

[0xced]

What version of xunit and xunit.runner.visualstudio are you using?

Also, what happens if you try with the Issue33 project from this branch on my fork?

[dArignac]

It is xunit 2.9.2 & xunit.runner.visualstudio 2.8.2 for me.
I am not sure how to build and use the code in your branch, tbh.

[0xced]

I am not sure how to build and use the code in your branch, tbh.

git clone https://github.com/0xced/Xunit.SkippableFact/
cd Xunit.SkippableFact
git checkout issue33

Then open Xunit.SkippableFact.sln in Rider and run the Issue33 test and post the same screenshot as I posted above.

[dArignac]

Thanks, I ran it. There it is alright.