Implement Pluggable agents

[CharliePoole]

@ChrisMaddock @mikkelbu @rprouse @jnm2

I'm calling this a question, but I mean it as a question we are asking ourselves.

Here's a rough outline of what I want to try with some spike code. Some of this is new ground so it would be nice to have all our brains on it.

FIRST, here's how things generally work. Except as noted, they are the same in both the current NUnit engine and the testcentric fork. Also, for clarity, I've simplified things a lot here.

1. When the user asks the engine for a runner, passing a package, it always gets a MasterTestRunner.
2. That runner is initially "bare" without any lower-level runners to back it up. But as soon as the tests need to be loaded, MasterTestRunner uses various services to fill in the appropriate internal runner (ITestEngineRunner).
3. If the ITestRunner is one of the in-process runners, it finds a driver (IFrameworkDriver) that knows how to run the tests. On platforms that support AppDomains, it creates one in which to run the test.
4. OTOH. if the runner is a ProcessRunner, things work differently. ProcessRunner uses the TestAgency service to get an agent. Although we usually just call them "agents", think of them as "process proxy agents" for this discussion, because the serve as an in-process proxy for the runner executing in a separate process.
5. TestAgency launches that process and waits for it to establish communication. Once it does it returns the proxy to the ProcessRunner.
6. The ProcessRunner issues commands to the remote runner and the remote runner goes through steps 1 to 3 itself.

The current GUI engine deviates from the above by

• Not supporting in-process execution - so step 3 never takes place.
• Supporting "pluggable agents", i.e. agents as extensions. However, they are still always proxies for a remote process, which they launch. (UPDATE: this is not yet merged to main)

Because the GUI engine has already taken those steps toward generalized agents, I'm using that engine for my spike.
UPDATE: See further comment on selection of a base for the spike.

SECOND, here's how I'd like to see agents work...

1. Continue to use a MasterTestRunner as we now do.
2. Rather than holding internal runners directly, MasterTestRunner will acquire an agent. A new service will be needed, which knows what agents are available and can decide which ones to use based on settings in the package.
3. An agent will return an internal runner on request. The engine will not actually know where that runner is located - whether in-process, in a separate process or on a separate machine.
4. Once `MasterTestRunner has a set of agents with runners, it will continue to call the runners directly, as it now does.
5. I'll wait to see what the code tells us, but I think we'll see some structural changes as a consequence of all this:
   • TestAgency will either disappear or be seriously restricted in what it does. The process management functionality will move into the particular agent type, for example.
   • We may end up distinguishing engine services from agent services, possibly leading to splitting the engine.core assembly.

FINALLY, if we decide that the spike answers the question in the title positively, I'll define a set of steps - independent PRs - to get it into the 4.0 release.

[rprouse]

Your proposal does sound 'cleaner' and looks like it simplifies supporting different runners without the complexity that we have now. I think that as you say, we'll need to see what the code tells us. I'd be interested in where this goes, but I will leave it up to @ChrisMaddock.

[CharliePoole]

I originally thought of using the GUI engine as a base for the spike. On reflection, that would be easy initially but would create more work at the end as everything would then have to be recreated in the NUnit engine.

That leaves the question of what we may want to do in the dev-4.0 branch before I fork the spike branch. Here's my plan, which I'll keep up to date as items are added or completed.

The spike was completed and the dev-4.0 branch is no longer used. This list now reflects changes being made to main

• Drop NUnitConfiguration class #912
• Remove DomainUsage options #913
• v4: Create single constructor method/structure for a TestPackage #889
• Eliminate In-Process Execution #861
• Refactor RuntimeFramework and related classes #1168
• Give agents individual names #877
• Modify arguments to agent process for 4.0 #955
• Replace ProcessRunner with agent launchers

[ChrisMaddock]

I have nothing to add, all this sounds great to me!

[CharliePoole]

See plan update above

[CharliePoole]

The spike having been completed, we are now in the process of working toward an actual 4.0 release. I'm renaming this to cover actual implementation of the feature.