How do you use and acquire TestPlatform?

[nohwnd]

Hello, I am from microsoft/vstest team. In the past we talked about how you acquire TestPlatform and how you use it. I looked through your code briefly, but I want to confirm my assumptions and that my info is up-to-date.

Acquisition

From what I gather you are either installing TestPlatform package to get vstest.console.exe or TestPlatform.Portable to get vstest.console.dll, you then run them using the TestPlatform.TranslationLayer (VSTestConsoleWrapper). Is that correct?

• Do you grab vstest.console from dotnet sdk installation if available?
  If yes:
  • Is running Stryker on system that has just .NET Runtime but not .NET SDK supported?
  • Is running Stryker on system that has only .NET Framework supported?

Usage and updates

I see from your Directory.Build.props that you are all up to date on the latest version 17.1.0. Are you generally moving to the latest TestPlatform version as soon as it is released? Or are you doing that once in a while, and I am just lucky to ask right after you updated?

[rouke-broersma]

Hi @nohwnd and thank you for asking these questions!

Acquisition

• We are using TestPlatform.Portable to get vstest.console.exe and vstest.console.dll because we run on both windows, linux and macos. On windows we use the exe and on other platforms we use the dll. I am not 100% sure why we don't use the dll in all cases, I suspect this is from before we deployed as a dotnet sdk tool, or because we had some issues with using the dll for dotnet framework tests.
• We use TestPlatform.TranslationLayer to drive vstest to perform testing yes
• The way we acquire TestPlatform.Portable is not very straightforward. This is because in modern nuget versions the tools folder does not get distributed with downstream packages. We distribute vstest.console in the following way:
  • We have a 'fake' project file that contains nothing but TestPlatform.Portable: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/ToolHelpers/vstest.fakeproject.csproj
  • In our azure pipeline we restore the fake project to a specified directory so the nuget package is resolved and downloaded to disk: https://github.com/stryker-mutator/stryker-net/blob/master/azure-pipelines.yml#L298
  • In our real project file we embed the nuget package as embedded resource so it is compiled with stryker: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/Stryker.Core.csproj#L37
  • We have a helper class that then extracts the embedded resource to disk so we can use it: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/ToolHelpers/VsTestHelper.cs#L179

Usage and updates

• We do not grab vstest.console from the dotnet sdk because we have noticed that there can be incompatibilities between TestPlatform.TranslationLayer and a specific vstest.console version

• Stryker cannot run on systems that do not have .NET SDK installed at this time, because we distribute stryker as a .NET SDK Tool. We do however have ideas for other runners such as a visual studio extension or perhaps from a webinterface (possibly blazor/client side).

• Stryker does work for .net framework projects, but .NET SDK is still required to run stryker at this time.

• We use dependabot to remain up to date with the TestPlatform packages, we have an integration test to make sure we can still deploy and drive vstest.console. This integration test is what caught the issue we had before that blocked us from upgrading (you closed this issue today because the problem is solved).

If you have any suggestions to make this more efficient I would love to hear it!

[nohwnd]

Thanks for the detailed response, and sorry about breaking you in microsoft/vstest#3144.

I am asking about this because I am working on a set of tests that ensure we are staying compatible with previous versions. And I am trying to maximize the value added, without adding way too much overhead.

If you have any suggestions to make this more efficient I would love to hear it!

I don't at the moment. But if you do, let me know! :)

I hope we will be able to stabilize our behavior to make translation layer work consistently with the version we ship in preview and stable dotnet sdk. That way you would be able to just grab vstest.console from there.

For my future reference, from what I see all your interactions with wrapper are done in this class: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/TestRunners/VsTest/VsTestRunner.cs And you do discovery and run with a custom non-debugging launcher.