Stryker.NET doesn't handle xUnit v3 properly

[0xced]

Describe the bug

When using xUnit v3, Stryker.NET produces warnings and can't perform proper mutations.

Reproduction steps

Checkout the xunit-v3 branch of the Serilog.Formatting.Log4Net project and run dotnet tool restore && dotnet stryker at the root of the repository.

Expected behavior

Stryker.NET performs mutations and gives the same score (100%) as when using xUnit v2.

Dashboard with xUnit v2 (on the main branch): 100%

Actual behaviour

Stryker.NET produces many warnings and eventually fails to perform the mutations.

[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.IndentationSettingsTest.IndentationSettingsToString), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.ThreadIdProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.MessageCDataMode), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.IndentationSettings), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.DomainAndUserNameProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.LogEventLevel), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.MachineNameProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.XmlElementsLineEnding), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.Log4JCompatibility), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.PublicApi.ApprovePublicApi), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.

Dashboard with xUnit v3: 3.09%

Desktop

• OS: both macOS and Linux (and probably Windows too)
• Type of project: .NET 6 + 8 + .NET Standard 2.0
• Framework Version .NET 8
• Stryker Version: 4.4.1

Additional context

I have not enabled Microsoft Testing Platform support in xUnit.net v3 so I don't think this is related to #3094 although I'm not totally sure.

[rouke-broersma]

We have quite some custom logic for dealing with specific test framework quirks, I suspect xunit v3 has added yet another layer of quirks.

[dupdob]

I predict a big one: it looks like a lot of VsTest design rules are violated here.

[dupdob]

First Analysis

Stryker is not able to run xUnit 3 theories with inline data, that is Stryker detects them correctly but they are skipped by the runner
TpTrace Warning: 0 : 18292, 12, 2024/11/22, 17:39:20.512, 224768786899, testhost.dll, [xUnit.net 00:00:00.22] Serilog.Formatting.Log4Net.Tests: Skipping test case 'Serilog.Formatting.Log4Net.Tests.IndentationSettingsTest.IndentationSettingsToString' (ID '6066754da9eb486a3555a1ff4792822d9ebe45d5ccf3d02486e8bdb36c7bed24') without serialization
it looks like Stryker should provide serialized data and somehow fails to do so.
In theory Stryker keeps all data for every test and reuses them afterward, but there is probably a bug somewhere.

The good news is that it does not look like a big issue.

[dupdob]

Update

• it looks like there are differences in how theories are captured between xUnit V2 and V3. Main (single?) impact is that theories data set are discovered at run time only
• there was a minor bug in Stryker regarding test cases discovered during the initial test run. Not sure if it has any actual impact. Fixing it did not solve the issue
• it looks coverage capture does not work at. The usual cause for this is failing to copy the mutated assembly in the right place
• mutated assembly is properly copied at the right place

Conclusion at this stage

It is likely that the xUnit V3 test runner executes the tests outside the VsTest host. But, Stryker requires its VsTest extension to run inside the test process.
Without this, Stryker simply cannot run effectively: it cannot get test coverage, nor control which mutant to activate during a test session.