`dotnet test` output text should make finding output for failed tests easy to find

[TheRealPiotrP]

Consider the following output of a dotnet test run:

  Failed   Microsoft.DotNet.Migration.Tests.GivenThatIWantToMigrateTestApps.It_migrates_project_with_output_name
  Results File: /Users/piotrp/code/github/piotrpMSFT/cli/test/dotnet-migrate.Tests/TestResults/_MacBook-Pro 2016-12-01 09:48:50.trx

  Total tests: 39. Passed: 33. Failed: 4. Skipped: 2.
  Test Run Failed.
  Test execution time: 3.2145 Minutes

The output is challenging to work with for a few reasons:

What Failed?

The line describing a failed test is Failed Microsoft.DotNet.Migration.Tests.GivenThatIWantToMigrateTestApps.It_migrates_project_with_output_name. When searching through a build log, there is not a string I can use which will uniquely identify these lines. xunit used [Failed] which is fairly unique. Some similar delimiters that are persistent and identify failed tests would be very useful.

Summary

The summary works well when test projects are running one at a time. When executing tests in parallel, it becomes difficult to see what test project the summary is for. A proposal:

 Test Run Summary
 -------------------
 Test Project: /path/to/some/test.csproj
 Results File: /path/to/some/results.trx
  
 Total tests: 39. Passed: 33. Failed: 4. Skipped: 2.
 Failed Tests:
   [Failed] My.Namespace.TestClass.Test1(some theory args)
   [Failed] My.Namespace.TestClass.Test2(some theory args)
   [Failed] My.Namespace.TestClass.Test3(some theory args)
   [Failed] My.Namespace.TestClass.Test4(some theory args)

 Test Run Failed.
 Test execution time: 3.2145 Minutes

/cc @livarcocc @blackdwarf

[blackdwarf]

Agreed. As it currently stands, the output of the dotnet test is not as good as Preview 2 output, even in the positive case (all tests are green).

[TheRealPiotrP]

Test1 -> [Failed] My.Namespace.TestClass.TheFailingTestMethod(some theory args)

[codito]

Triage:

• Use colors for pass/fail result. See AnsiColor in dotnet-cli repo. Check with msbuild if we could piggyback their colorization.

Skipped tests -- Yellow

[Failed]My.Namespace.TestClass.Test1(some theory args) -- RED
Stacktrace -- RED

• Update console logger to enhance summary as follows:

 Test Run Summary
 -------------------
 Test Project: /path/to/some/test.csproj
 Results File: /path/to/some/results.trx
  
 Total tests: 39. Passed: 33. Failed: 4. Skipped: 2.
 Failed Tests:
   [Failed]My.Namespace.TestClass.Test1(some theory args)
   [Failed]My.Namespace.TestClass.Test2(some theory args)
   [Failed]My.Namespace.TestClass.Test3(some theory args)
   [Failed]My.Namespace.TestClass.Test4(some theory args)

 Test Run Failed.
 Test execution time: 3.2145 Minutes

• Same string [Failed]My.Namespace.TestClass.Test1 should be in the log. User could use it to search through the log.
• Check if adapter provides the right TestMethod title

[TheRealPiotrP]

AnsiConsole in CLI repo

[natemcmaster]

Does this also mean you won't be showing a line for each successful test by default? On large tests, the output of vstest console is overwhelming. (Example: ASP.NET Core's test project has ~3000 tests... x2 because we run on .NET Core and .NET Framework).

Can't tell if this is part of the triage decision here. If not, should I open another issue?

[codito]

@natemcmaster #301 tracks minimizing console output in test runs.

[codito]

We need to decide how this summary plays with minimal verbosity #301.