An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error when using Real Time Discovery

[loop-evgeny]

When reporting a bug, please provide the following information to speed up triage:

• NUnit 3.13.2, NUnit3TestAdapter 4.1.0 (also happens on 4.2.0-beta.4)
• Visual Studio 2019 (16.11.8) and 2022 (17.0.4)
• .NET Core 3.1

Repro: NunitFSharpTestCaseSource.zip

I have a C# test project with a TestCaseSource that uses an F# type (discriminated union). When I try to run all tests using the VS test runner no tests get run and the Tests output window shows

========== Starting test discovery ==========
NUnit Adapter 4.1.0.0: Test discovery starting
NUnit Adapter 4.1.0.0: Test discovery complete
========== Test discovery finished: 1 Tests found in 1.3 sec ==========
========== Starting test run ==========
NUnit Adapter 4.1.0.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Invalid Condition 'FullyQualifiedName=CSharpTest.MyTests.RunSomeTests
  BofDU3 = DU3Member "bbb" }'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscoveryContext.GetTestCaseFilter(IEnumerable`1 supportedProperties, Func`2 propertyProvider)
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_TfsTestCaseFilterExpression() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 100
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_IsEmpty() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 102
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.InitializeForExecution(IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 248
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 105
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 41 ms ==========

I can run the test using dotnet test from the command line, however.

This looks like bug #691 which was closed a while ago.

[OsirisTerje]

@loop-evgeny I am not able to reproduce this. I am running your repro on VS2022 (17.0.2)
and I get this:

and with no error message as you got.

[loop-evgeny]

Hmm, that's weird - I also tried it on another machine (VS 2019) and it worked there. So it seems to be something specific to my machine. How can I debug this further?

[OsirisTerje]

Can it be some culture-setting? It fails in the parsing check in vstest.console, and it is very sensitive to the different characters being used. I am thinking of the '{', '(', '='. If any of these are off then that might have something to do with it.
If you clone and build the vstest platform project, and then check your expression there, you can see if it pass that filter. Look in the FilterExpression.cs, method Parse. I added a test for a similar issue in the FilterExpressionTests.cs.
Make the Parse method public and turn off the signing in all the projects, make life easier :-)
I just checked my expression like this:

[TestMethod]
        public void ParsingShouldHandleExpressionWithParanthesis()
        {
            var filter = @"FullyQualifiedName=Issue919.Foo.Baz\(1\)";

            var result = FilterExpression.Parse(filter,out FastFilter fastFilter);
            
        }

I debugged this one, and checked it for failure. It should fail here too for you, and you could then debug in and see what it complains about.

Now, since I could clone your project and it worked "as is", the expression itself should be ok - no culture issues in it. Also since it works with dotnet test on your machine, it looks like VS is causing this.

You can also debug into the adapter, there is information about that here, but I don't think you find anything more there, since the real error pops up from the vstest platform.

Also check that your setting for VS doesn't follow your machine, See Options/Environment/Internation Settings.

You can also turn on different tracings, but I don't think your get any closer with that.

When you figure this out, please drop a comment on that here too. :-)

[loop-evgeny]

if you clone and build the vstest platform project, and then check your expression there

It fails to build, both using VS 2022 and the command line. The problem is not obvious and that's another whole rabbit hole I don't really want to go down. But do you get the same FullyQualifiedName generated as the one in my error message?

FullyQualifiedName=CSharpTest.MyTests.RunSomeTests
  BofDU3 = DU3Member "bbb" }

[OsirisTerje]

This is a bit weird.

The error string you show is not a valid filter expression.

The TestCaseFilter should not have been invoked here at all. You're running from the Test Explorer, using the Run All there, and it should just give you list of test cases, in this case only one. No TestCaseFilter should be used at all.
I just debugged through the adapter on this, and there is no invocation of any TestCaseFilter here.

You could do the same on your machine, and see where it goes. Use the debug link for the adapter above. To make it simpler, I've enclosed a debug version of the adapter here, and you just need a .runsettings file added to your solution, containing:

<RunSettings>
   <NUnit>
       <DebugExecution>true</DebugExecution>
   </NUnit>
</RunSettings>

and of course, clone the source code for the adapter, and load that into a VS.

Also, the error message you get doesn't match up with what one would expect, there is no leading braces there, it seems like it is truncated - but - anyway - it shouldn't even get there.

--> Also, in the same .runsettings file, set NUnit.Verbosity to 5, you should then see what expressions it looks for. ACtually, start with just setting the verbosity, and see what comes out. One thing you should see is the Runtype, which should show Ide.
NUnit3TestAdapter.4.2.0-dbg.zip

[loop-evgeny]

I'm sorry, I feel really dumb, but I just cannot get it to hit any breakpoint in NUnit3TestAdapter code. I managed to build the debug Nuget package with build -c debug; build -t package -c debug and managed to install that into my test project. I opened NUnit3TestAdapter.sln in another VS 2022 instance and attached the debugger to the VS instance with my test solution. If I select "Automatically determine the type of code to debug" or select "Managed (.NET 4.x)" when attaching it shows messages in the Debug output window about loading many assemblies, but not NUnit3.TestAdapter.dll. If I select any other process type (.NET Core, .NET 3.x, Native, whatever) I get no messages at all in the Debug output window. In both cases breakpoints I set do not get hit. Sometimes they show the "failed to bind" icon, sometimes they show the filled-in red circle, implying that the breakpoint "bound" - but it still does not get hit. I tried setting breakpoints at, for example, NUnit3TestExecutor.RunTests and VsTestFilter.TfsTestCaseFilterExpression (both appear in the stack trace).

[loop-evgeny]

I do see the error message in the Debug window of the VS instance with NUnit3TestAdapter.sln, so I've attached to the right process:

Informational: Starting test discovery for requested test run
Informational: ========== Starting test discovery ==========
Informational: NUnit Adapter 4.2.0.0: Test discovery starting
Informational: NUnit Adapter 4.2.0.0: Test discovery complete
Informational: ========== Test discovery finished: 1 Tests found in 962 ms ==========
Informational: ========== Starting test run ==========
TableControl: QueueUpdate 28/12/2021 5:50:36 PM:234
TableControl: About to update entries 28/12/2021 5:50:36 PM:342
TableControl: Updating entries 28/12/2021 5:50:36 PM:344
TableControl: Updating entries (UI thread) 28/12/2021 5:50:36 PM:348
TableControl: Refreshing UI 28/12/2021 5:50:36 PM:349
Informational: NUnit Adapter 4.2.0.0: Test execution started
Error: An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Invalid Condition 'FullyQualifiedName=CSharpTest.MyTests.RunSomeTests
  BofDU3 = DU3Member "bbb" }'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscoveryContext.GetTestCaseFilter(IEnumerable`1 supportedProperties, Func`2 propertyProvider)
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_TfsTestCaseFilterExpression() in C:\W ork\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 100
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_IsEmpty() in C:\W ork\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 102
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.InitializeForExecution(IRunContext runContext, IFrameworkHandle frameworkHandle) in C:\W ork\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 254
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in C:\W ork\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 104
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

Informational: ========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 85 ms ==========
TableControl: QueueUpdate 28/12/2021 5:50:37 PM:409
TableControl: About to update entries 28/12/2021 5:50:37 PM:510
TableControl: Updating entries 28/12/2021 5:50:37 PM:513
TableControl: Updating entries (UI thread) 28/12/2021 5:50:37 PM:544
TableControl: Refreshing UI 28/12/2021 5:50:37 PM:545

[OsirisTerje]

You don't have to attach the adapter VS to your solution. Just let it stay there.
In your Test SOlution, where you have added a packagereference to the dbg version (it should end in -dbg), you just start the test.
Note; You MUST have the .runsettinsg file there with the DebugExtension set to true, ensure it is actually active.

Then just run the test (don't debug - just run it), and a dialog box should appear:

And you end up in a method in the adapter code called CheckDebug. Step out of that, you're in the code.

[loop-evgeny]

OK, it turns out the problem was I hadn't selected the .runsettings file. When I do that it launches the debugger when I try to run tests and I'm able to hit a breakpoint in VsTestFilter.IsEmpty. (For some reason it refuses to use the existing attached debugger and insists on opening a new VS instance every time, which is quite annoying. How do I get it to keep using the debugger that's attached?)

I'm still not sure how to debug this though. Is the FullyQualifiedName name wrong (e.g. it should not have { in it)? Then where is that generated? Or should I be trying to figure out why it's calling VsTestFilter.IsEmpty at all?

[OsirisTerje]

See post above :-) Just have the VS open with the adapter code, and choose that one

Don't attach

[OsirisTerje]

Yes, the FQN is wrong

But, you shouldn't even have one there.

(I'm updating now to 17.0.4 just to be sure there is nothing changed there. Should not since it is a patch version)

Just checking:
You start this using the Run All button (leftmost) in Test Explorer ?
There is nothing in your Search box (shouldn't matter though) ?
If you select the single test there and run that, does it still fail ?
If you add one more test, just a simple one with a single Assert.Pass() in it, does that invoke the same error?
If you add a TestCase with a string parameter and start with a very simple string (like "aaa"), does that work ?

[OsirisTerje]

Looking at your stack trace, you're getting in at the RunTests(IEnumerable<string> sources method, all I get is that I get in at the RunTests(Ienumerable<TestCase> tests method. The latter is the one that is used with Test Explorer, and you should not see the call to the first one. You're doing something special here....
Can you describe in detail (pref with screenshots) how you start the test? And it is VS 2022 with the exact same repro, right ?

[OsirisTerje]

Just a small tip: In case there is any older versions of Visual Studio testhost lingering about, you should clear the mefcaches, they are located for each version under e.g.

%localappdata%\Microsoft\VisualStudio\14.0\ComponentModelCache

Go through all version you find there, and delete all these caches. They will rebuild, but there has been cases where these have been used by newer versions.

[loop-evgeny]

I have nothing in the Search box.

I start the test by initially having Test Explorer closed and choosing Test, Run All Tests from the main menu:

So it turns out that the test did run, despite the error, but VS Test Explorer was somehow set to show only "not run" tests by default, so this was what I saw:

Combined with the error in the Tests output window this led me to believe the test did not run. But if I get it to show everything:

If you select the single test there and run that, does it still fail ?

If I select "RunSomeTests({ AofDU2 = DU2MemberOfDU1 (DU1Member "aaa")
BofDU3 = DU3Member "bbb" })" in Test Explorer I can run that. "RunSomeTests" does not run, however.

If you add one more test, just a simple one with a single Assert.Pass() in it, does that invoke the same error?

That runs fine.

So, in summary, it seems that this particular error does not prevent the tests from running, it just adds an extra "Not run" test and a misleading error.

In my real test project however, even if I comment out the test that causes this error I still get another error:

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

... and no tests run! How do I find out which test causes this?

There is no stack trace displayed in this case, but attaching the debugger shows that it happens in the same place:

         public ITestCaseFilterExpression TfsTestCaseFilterExpression =>
            testCaseFilterExpression ??= runContext.GetTestCaseFilter(SupportedProperties, PropertyProvider);

If I select a specific test in Test Explorer (in my real project) I can run that. However, the "Run all tests in view" button does not run anything.

I tried deleting all the ComponentModelCache directories I found there, but that didn't help.

[OsirisTerje]

1. The extra node you see there, RunSomeTests, comes from the Real Time Test Discovery, and is not a valid result. The RTD can't figure out the underlying tests since it just does source discovery. I would turn this option off, as it is just confusing in your case,
   

2. It should not be a single test that causes this. If you add the dbg version of the adapter to your real test code, you should be able to single step through it the same way. That method is being used from different places in the adapter code, but since it stopped on the InitializationForExecution code in the repro, it should stop in the same place for your real code.

3. I have asked MSPG about which situations where the sources interface method could be called.

[loop-evgeny]

I managed to track down the second error ("Missing '('.") by selecting subsets of tests to run. Here is a repro for it (I added this to the ZIP in the original issue description, too):

        private static IEnumerable<TestCaseData> GetBadTest2Data()
        {
            yield return new TestCaseData(new[] { ("ccc", "ddd") });
        }

        [TestCaseSource(nameof(GetBadTest2Data))]
        public void BadTest2(IEnumerable<(string, string)> collection) => throw new NotImplementedException("If you get this exception BadTest2 ran!");

Unchecking "Discover tests in real time ..." allows all tests to run, including in my real test projects! Thank you so much for your help!

So that fixes the problem for me, but of course it's quite a "gotcha"!

[OsirisTerje]

Thanks a lot for your persistence! It can confirm this, it breaks on my site too when I turn on the "Discover tests in real time".

I'll report this to the MSPG too, and you're right, it's quite a "gotcha" !

Thanks again and a Happy New Year !