NUnit Console - "NUnit.Engine.NUnitEngineException: Remote test agent believes agency process has exited"

[fandrei]

NUnit Console 3.16.3
Started getting these errors, not sure why:

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    _Bin.Testing\Tests.dll

Unhandled Exception: NUnit.Engine.NUnitEngineException: Remote test agent believes agency process has exited.
   at NUnit.Engine.Services.TestAgency.OnAgentExit(Process process, Guid agentId)
   at NUnit.Engine.Services.TestAgency.<>c__DisplayClass13_0.<GetAgent>b__0(Object sender, EventArgs e)
   at System.Diagnostics.Process.OnExited()
   at System.Diagnostics.Process.RaiseOnExited()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)

Errors, Failures and Warnings

1) Error :
System.Net.Sockets.SocketException : An existing connection was forcibly closed by the remote host
--SocketException
An existing connection was forcibly closed by the remote host
   at NUnit.Engine.Communication.Transports.Tcp.SocketReader.GetNextMessage()
   at NUnit.Engine.Communication.Transports.Tcp.TestAgentTcpProxy.TestRunResult(ITestEventListener listener)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2023-02-27 02:57:47Z
    End time: 2023-02-27 02:57:50Z
    Duration: 2.420 seconds

[mikkelbu]

Transferring this to the nunit-console project.

@fandrei Can you provide some more information. The console project has the following "template" and can you also say which version of .NET you are using (the last item below only makes sense on .NET framework)

To help the others diagnose and reproduce your issue, please include as many of the following as relevant:

• The full command line being used
• Method of installation (e.g. NuGet, zip, msi)
• Log files (Run the console with --trace=Debug to produce these)
• Version of the NUnit Engine/Console
• All versions of the NUnit Framework in use
• Whether the behaviour is any different when the tests are run using the --inprocess flag

[fandrei]

@mikkelbu

• nunit3-console.exe _Bin.Testing\Tests.dll
• NuGet
• the logs contain nothing unusual except these errors:

18:57:49.508 Debug [ 1] TestAgency: Waiting for agent {5d793f8f-1878-4f6f-8e37-533f37229d56} to register
18:57:49.723 Debug [ 1] TestAgency: Returning new agent {5d793f8f-1878-4f6f-8e37-533f37229d56}
18:57:51.848 Error [ 1] ProcessRunner: Failed to run remote tests --SocketException
An existing connection was forcibly closed by the remote host
   at NUnit.Engine.Communication.Transports.Tcp.SocketReader.GetNextMessage()
   at NUnit.Engine.Communication.Transports.Tcp.TestAgentTcpProxy.TestRunResult(ITestEventListener listener)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

18:57:51.854 Info  [ 1] ProcessRunner: Unloading 
18:57:51.854 Warning [ 1] ProcessRunner: Failed to unload the remote runner. --SocketException
An existing connection was forcibly closed by the remote host
   at NUnit.Engine.Communication.Transports.Tcp.TestAgentTcpProxy.SendCommandMessage(String command, Object[] arguments)
   at NUnit.Engine.Runners.ProcessRunner.UnloadPackage()

18:57:51.854 Error [ 1] ProcessRunner: System.Net.Sockets.SocketException : An existing connection was forcibly closed by the remote host
18:57:51.854 Error [ 1] ProcessRunner: --SocketException
An existing connection was forcibly closed by the remote host
   at NUnit.Engine.Communication.Transports.Tcp.TestAgentTcpProxy.SendCommandMessage(String command, Object[] arguments)
   at NUnit.Engine.Runners.ProcessRunner.UnloadPackage()
   at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)

18:57:51.870 Info  [ 1] ServiceManager: Stopping DefaultTestRunnerFactory
18:57:51.870 Info  [ 1] ServiceManager: Stopping ResultService
18:57:51.870 Info  [ 1] ServiceManager: Stopping DriverService
18:57:51.870 Info  [ 1] ServiceManager: Stopping TestAgency
18:57:51.871 Info  [ 1] ServiceManager: Stopping RuntimeFrameworkService
18:57:51.871 Info  [ 1] ServiceManager: Stopping ProjectService
18:57:51.871 Info  [ 1] ServiceManager: Stopping ExtensionService
18:57:51.871 Info  [ 1] ServiceManager: Stopping TestFilterService
18:57:51.871 Info  [ 1] ServiceManager: Stopping RecentFilesService
18:57:51.872 Info  [ 1] ServiceManager: Stopping SettingsService

• all versions are 3.16.3
• NUnit used in the project is 3.13.3
• with --inprocess I'm getting this error:

NUnit.Engine.NUnitEngineException : The NUnit 3 driver encountered an error while executing reflected code.
  ----> System.InvalidCastException : Unable to cast transparent proxy to type 'System.Web.UI.ICallbackEventHandler'.

--NUnitEngineException
The NUnit 3 driver encountered an error while executing reflected code.
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
--
InvalidCastException
Unable to cast transparent proxy to type 'System.Web.UI.ICallbackEventHandler'.
   at NUnit.Framework.Api.FrameworkController.LoadTestsAction..ctor(FrameworkController controller, Object handler)

[fandrei]

PS running the tests from Visual Studio fails, too.

[CharliePoole]

@fandrei In the title you say "believes." Do you have reason to believe that the agent process may not have exited?

[fandrei]

@CharliePoole it doesn't matter what it believes. But all the tests weren't run.

[CharliePoole]

@fandrei Oh I see... the word comes from the exception message itself. The agent is unable to connect to the engine (agency) process, so it exits giving a return code, which translates to the message you see. Of course, the agency hasn't exited. Otherwise it would not be able to report the message to you. :-) So... a TCP communication error, apparently at or prior to the loading the test - otherwise, more info would be received and shown in the logs.

[fandrei]

@CharliePoole the message that says "Unable to cast transparent proxy to type 'System.Web.UI.ICallbackEventHandler'" looks suspicious.

[CharliePoole]

Indeed it does. You haven't said what runtime you are targeting. System.Web.UI.ICallbackEventHandler is only used when the console runner is running a test targeting the .NET Framework. Use of TCP indicates that it's running a .NET core or 5.0+ test.

Info still needed...

• Runtime targeted by your test assembly (as asked by @mikkelbu previously)
• Where you got the copy of nunit3-console.exe you are running, i.e., what package you installed and where the exe is located in the directory structure created by that package. I'm suspecting a confusion between the .NET 6.0 build of nunit3-console.exe and the .NET Framework 4.6.2 build. The former cannot run .NET Framework tests, by design.

CORRECTION: I see that the message about System.Web.UI.ICallbackEventHandler only appeared with the --inprocess option. That's the error we would expect if you try to run a .NET Core test assembly in-process under .NET Framework. Probably, a better message would have helped there.

[fandrei]

@CharliePoole my project targets .NET 6.
As much as I remember I used nuget.exe to download the console runner. I wonder if there is any recommended way to do it for .NET 6 projects.

[CharliePoole]

What you are using is correct. The engine launches a process under .NET 6.0. That process is crashing. Did it work in earlier versions of the console runner and engine? E.g. 3.16.2?

[fandrei]

@CharliePoole it all worked until recently, with the same versions. Maybe it's a system config problem. But I have no idea why it happened and how to fix it.

[CharliePoole]

It's possible that some change to your tests is putting a demand on the engine, which was not there before. I suggest trying an earlier version of the source from your repository.

[fandrei]

@CharliePoole it's happening for all the projects that I've tried so far.

[OsirisTerje]

@fandrei Do you say that version 3.15.2 works, but not 3.16.3?
If so, could this be another fail-out from #1324 ?

[rprouse]

@fandrei could you try the previous release of the Console and see if it works? That will help us track down if it is a regression in the latest release.

Thanks

[fandrei]

@OsirisTerje no, they both don't work now.

[fandrei]

It's really strange, but I'm still getting same errors even after setting up Visual Studio on my new PC from scratch. With runner 3.15.2, too.

[OsirisTerje]

@fandrei Are you able to create a small repro for this that you can upload?

[fandrei]

Apparently, this was a strange case of antivirus interaction.