NUnit.Engine: Exploring test assembly tree returns result with a failure message

[i-m-luke]

I'm developing a custom testrunner app and I want to display a test tree structure. I want to read the test tree structure directly from a test assembly file and to achieve that I use this code:

            ITestEngine engine = TestEngineActivator.CreateInstance();
            TestPackage testPackage = new TestPackage("... test assembly path ...");
            ITestRunner runner = engine.GetRunner(testPackage);
            XmlNode tree = runner.Explore(NUnit.Engine.TestFilter.Empty);

I sucessfully get a result from the Explore method but there is a failure message in the result xmlnode and no desired test tree structure. Here is the failure message from the result: " System.ComponentModel.Win32Exception :The system cannot find the file specified. "

After some experiments I've figured out the conditions that have to be met to explore tree stucture sucessfully:

• Project (custom testrunner app) and explored test assembly must have the same target framework (in my case .NET Framework 4.6.1)
• Project must have a refrence to the explored test assembly

Is this a feature or a bug? If it's a feature: Is there any way to workaround this? Because I'd like to use some later framework (.NET Framework 4.8 or .NET) as target framework of the application. Yes, I could higher up target framework version of the test assembly but I'd rather avoid this solution.

[OsirisTerje]

Have you looked at the code for the NUnit.Engine? I'm transferring your question to that repo.
If you follow the same way it is done there, also, have a look at the adapter code. The adapter is a runner, so you could possibly get some ideas there.

[i-m-luke]

@OsirisTerje So I've managed to get the test tree stucture using NUnitTestAssemblyRunner. When the test assembly has .NET Framework 4.61 and the project I'm getting the structure from has 4.8.1 then I get the structure sucessfuly from the ITestAssemblyRunner Load method. But if the project has .NET 5.0 SDK then the result from the Load method has no tests in the Tests property.

[OsirisTerje]

Check if the "software under test" (SUT) works with dotnet test. If it does, then you have missed something in your runner. If not, then you have something in your SUT that doesnt match up with the NUnit requirements.

[OsirisTerje]

And, just curious: Are you creating your own runner for fun/training or is there some special situation you are trying to solve?

[i-m-luke]

And, just curious: Are you creating your own runner for fun/training or is there some special situation you are trying to solve?

We run our UI tests on multiple environemnts that doesn't have VS installed. Since the beginning, we've been running tests with NUnit Console Runner but this solution has become not suitable for our current needs. Sometimes we have to rerun some testcases and for this case it would be much better to have a tree structure a tester can choose the testcases from.

[CharliePoole]

Have you tried the TestCentric GUI runner?

[i-m-luke]

@CharliePoole I have not, but I definitely will. Altough I'm not sure wheter it will cover all of our needs (e.g. we need to setup the environment based on settings set by a tester before the test, generate result file with specific structure and make an API call after the test)

[i-m-luke]

Check if the "software under test" (SUT) works with dotnet test.

@OsirisTerje Could you please be more specific? If you meant to run the test with another test adapter then I can verify it's possible to run it with Visual Studio Test Adapter.

[CharliePoole]

@i-m-luke Key question for me is the version and target runtime of the engine you are getting from TestEngineActivator. That will depend on how you have installed the engine. The engine targets either .NET 4.6.2 or .NET Standard2.0 and the two builds work quite differently.

[CharliePoole]

@i-m-luke Try this with the latest build of the engine from our myget feed. It has targets for both .Net 6.0 and 8.0.