nunit3-console doesn't see nugets from Microsoft.AspNetCore.App

[Harmyder]

I have the following project and test

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.0" />
    <PackageReference Include="NUnit" Version="3.13.3" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
  </ItemGroup>

  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>
  
</Project>

[Test]
public void Test1()
{
    Assembly.Load("Microsoft.AspNetCore, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60");
}

I can run fine this project from Visual Studio but then I run it from command line with nunit3-console nunittests.dll I see the following error

System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.AspNetCore, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
at System.Reflection.Assembly.Load(String assemblyString)
at NUnitTests.Tests.Test1() in D:\Yola\Test22\NUnitTests\UnitTest1.cs:line 16
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

If I use Assembly.LoadFrom with the full path it is able to load the DLL.

[OsirisTerje]

Hi!

We do have issues with this kind of loading behavior, and we do have multiple attempts at finding the reason.
I made small repro solution, I got a different error from what you have, but I believe it might be about the same bug. The repro is here: https://github.com/nunit/nunit.issues/tree/main/Issue4654 . Feel free to change it.

I do get this error:

We do have multiple similar errors in the console repo, see https://github.com/nunit/nunit-console/issues

I'll move this issue over to that repo.

You can have a look at #1383 which is a similar reported issue.

See this comment:

The reason is that the nunit-agent only references the Microsoft.NETCore.App runtime in its runtimeconfig.json. Therefore a workaround is to manually add the missing runtimes (Microsoft.WindowsDesktop.App and/or Microsoft.AspNetCore.App) to the runtimeconfig.json.

If you use the latest test.sdk, you should also get the runtimeconfig.json correct, and then you should get the same error as I have above. See the repro for details.

The failure with loading Microsoft.Extensions.Logging.Abstractions is a bit weird. It is a framework reference, and you see it in the dependencies list in Visual Studio, but the console runner doesn't find it.

We have to dig a bit deeper to see what this can be.

Is there a reason you can't use dotnet test instead of the console?

[bthharper]

Using "nunit-console3" to run unit tests for .NET 8.0 assemblies that require the "Microsoft.AspNetCore.App" framework fails with the following error:

Results (nunit3) saved as TestResult.xml
PS C:\git\common> C:\Platform\nunit.console\net8.0\nunit3-console.exe C:/git/common/Source/Framework.Test/bin/x64/Debug/Framework.Test.dll
NUnit Console 3.17.0+685c5b542b5e9ba632c905f0bd514a773d9758af (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
17 May 2024 23:56:17

Runtime Environment
   OS Version: Microsoft Windows 10.0.22631
  Runtime: .NET 8.0.5

Test Files
    C:/git/common/Source/Framework.Test/bin/x64/Debug/Framework.Test.dll


Errors, Failures and Warnings

1) Invalid : C:/git/common/Source/Framework.Test/bin/x64/Debug/Framework.Test.dll
Unable to load one or more of the requested types.
Could not load file or assembly 'System.Configuration.ConfigurationManager, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
  ----> Could not load file or assembly 'System.Configuration.ConfigurationManager, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.Microsoft.AspNetCore.App

The workaround was to change the nunit3-comsole.runtimeconfig.json in the net8.0 folder to:

{
  "runtimeOptions": {
    "tfm": "net8.0",
    "frameworks": [
      {
        "name": "Microsoft.NETCore.App",
        "version": "8.0.0"
      },
      {
        "name": "Microsoft.WindowsDesktop.App",
        "version": "8.0.0"
      },
      {
        "name": "Microsoft.AspNetCore.App",
        "version": "8.0.0"
      }
    ],
    "configProperties": {
      "System.Reflection.NullabilityInfoContext.IsSupported": true,
      "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization": false
    }
  }
}

I would rather not have to modify the distribution of nunit3-console to make this work if possible, is there another solution I have missed for this?

As an aside, I am unable to use dotnet test as this does not work when the solution contains C++/CLI vcxproj - which is something Microsoft seem reluctant to fix.

[CharliePoole]

@Harmyder I added your load test to our existing package (integration) test for AspNETCore. All tests continue to pass in main. Can you try the same test using the latest dev build from our MyGet feed?

The FrameworkReference in your project is redundant in my tests and gives a warning message. However, I have some other tests, which create dependencies on the framework, so you might actually need it in the case of your example.

[CharliePoole]

@bthharper Your comment is a bit general and may call for a separate issue with more specific info. However, I'd suggest you also try the latest build on MyGet to see if it works.

I agree that modifying the runtime config for the console or agents is not satisfactory. It's also not possible for us to do it, since that would mean that nobody could use the console runner unless they had all those frameworks referenced installed, which is not always the case in certain test situations.

BTW, since you fixed your problem by changing the runtime config for the console itself, I deduce that you are using the .NET Core version of the runner rather than the standard runner. If you file an issue, please be sure to specify that to avoid confusion. I'm afraid you can't rely on me to remember it, as I'm pretty much stateless when working bugs. :-)

[bthharper]

@CharliePoole - thanks for replying, I can raise a separate issue if that would help.

To clarify, I unit testing .NET 8 (so .NET Core) assemblies against nunit.console\net8.0\nunit3-console.exe.

I will try the latest build and see if that helps.

[CharliePoole]

@Harmyder
Are you also using the net8.0 build of nunit3-console.exe? That's important since the .net 8.0 build has significant functional differences from the standard runner.