Batch (console) runner for TestCentric

[CharliePoole]

TestCentric is currently the only standalone runner, which is able to run both .NET Core and .NET Framework tests together, in one run, and combine the results.

This makes having a batch runner, which could be used in a CI environment a very attractive proposition.

The GUI itself is already run in our own CI tests. However, this somewhat wasteful of resources in the general case. A batch runner would essentially just be a different front end, displaying only to console and saving a result file as is now done.

[CharliePoole]

Initial "specs" for the console runner...

Program Name

Executable is tc-console.exe. May also create tc.exe or an equivalent shell command to invoke it (see Command Syntax below).

Features

• Run the tests specified in the command-line. Display an error message and return an error code if none are provided, otherwise, return an NUnit-compatible result code. Save result to TestResult.xml in the current directory by default, using nunit3 format.

• Explore the tests specified in the command-line. Display an error message and return an error code if none are provided, otherwise, return an NUnit-compatible result code. Save result to TestResult.xml in the current directory by default, using nunit3 format.

• Display or modify TestCentric settings from the command-line.

• List extension points and installed extensions. Add, remove, enable and disable extensions.

• Display the version of the runner and engine in use.

• Display help messages.

Command Syntax

Two basic options...

1. Files and options similar to NUnit. Examples...

   • tc-console mytests.dll
   • tc-console mytests.dll --explore

2. Sub-Commands similar to git or dotnet (I'm inclining toward this option)

   • tc-console [run] mytests.dll // run sub-command is default if no command is given
   • tc-console explore mytests.dll

Note: In either case, we might allow use of tc rather than tc-console, e.g.: tc explore mytests.dll

Sub-Commands

If we go the sub-command route, the initial list I'm considering is...

• run - to run tests
• explore - to explore tests
• extensions - to list, add, remove, enable and disable extensions.
• settings - to list, add, update and remove application settings
• version - to display version information
• info - to display extended info (version + settings + extensions)
• help - to display help text

[CharliePoole]

@TestCentric/gui-team Any thoughts on this?

[immeraufdemhund]

is there an option to continue watching the assembly and rerun all the tests like you would with npm test or simlar. I like it when I can just save my file and it runs my tests. It's harder for compiled tests I guess.. but that used to be an option right?

[CharliePoole]

It was an NUnit V2 GUI option and it is still an option for the TestCentric GUI. It makes sense in a GUI, since the program is kept open all the time.

When a console runner is used, running on change is generally done by a separate watcher program, which launches it when it sees a change. While it could be done internally, I'm not sure I see the advantage.