VSTest retry functionality failes when having spaces or ampersands (&) in TestCase parameter

[SabotageAndi]

When reporting a bug, please provide the following information to speed up triage:

• NUnit and NUnit3TestAdapter versions:
  NUnit: 3.12
  NUnit3TestAdapter: 3.15.1
• Visual Studio edition and full version number (see Help About)
  VS 2019 Enterprise Version 16.8.1
• A short repro, preferably attached or pointing to a git repo or gist
  https://github.com/SabotageAndi/NUnitRetry2213
• What .net platform and version is being targeted
  .NET Core 3.1
• If TFS/VSTS issue, what version, hosted, on-premises, and what build task you see this in
  Azure DevOps/ VSTest Task V2

The Azure DevOps VSTest task has the functionality to rerun failed tests. It looks like it doesn't work when you are using TestCaseAttributes and have spaces or ampersands (&) in the parameters.

Error:

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=NUnitRetry2213.Features.CalculatorFeature.AddTwoNumbers(\"Spa ces\"\,null)"
"D:\a\1\s\NUnitRetry2213\bin\Release\netcoreapp3.1\NUnitRetry2213.dll"
/Settings:"D:\a\_temp\0lawbeaogzy.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"D:\a\1\s"
##[error]Unhandled Exception: System.ArgumentException: Illegal characters in path.
##[error]   at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)
##[error]   at System.IO.Path.IsPathRooted(String path)
##[error]   at vstest.console.Internal.FilePatternParser.GetMatchingFiles(String filePattern)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.CommandLineOptions.AddSource(String source)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.TestSourceArgumentExecutor.Initialize(String argument)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
##[error]   at System.Lazy`1.CreateValue()
##[error]   at System.Lazy`1.LazyInitValue()
##[error]   at System.Lazy`1.get_Value()
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
##[error]   at System.Lazy`1.CreateValue()
##[error]   at System.Lazy`1.LazyInitValue()
##[error]   at System.Lazy`1.get_Value()
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.GetArgumentProcessors(String[] args, List`1& processors)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)

Code:

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Add two numbers")]
[NUnit.Framework.CategoryAttribute("mytag")]
[NUnit.Framework.TestCaseAttribute("Spa ces", null)]
public virtual void AddTwoNumbers(string column, string[] exampleTags)

The code is generated by SpecFlow, but I think we are generating the code correctly. We could escape the string in SpecFlow when we create the code, but there would nowhere be a normalization of it.

Repro project: https://github.com/SabotageAndi/NUnitRetry2213
GitHub Issue at SpecFlow: SpecFlowOSS/SpecFlow#2213

For me, it looks like there is some escaping missing in the test adapter full qualified name generation for spaces.

[OsirisTerje]

@SabotageAndi There are known issues around FullyQualifiedNames so this can be another one of those. But, I don't see the adaptercode in the stack trace there, so wonder a bit if this is something in the retry code. A bit strange also that it only happens on retry. If it was the adapter it should have triggered also on the initial run. Can you check more?. E.g. turn on verbosity in the runsettings file, and see if you get anything from the adapter at all.

[SabotageAndi]

The retry functionality in the tasks works, that it is starting a new vstest.console.exe call with a TestCaseFilter parameter.

This is the second call.

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=NUnitRetry2213.Features.CalculatorFeature.AddTwoNumbers(\"Spa ces\"\,null)"
"D:\a\1\s\NUnitRetry2213\bin\Release\netcoreapp3.1\NUnitRetry2213.dll"
/Settings:"D:\a\_temp\0lawbeaogzy.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"D:\a\1\s"

This is the generated filter parameter:

/TestCaseFilter:"FullyQualifiedName=NUnitRetry2213.Features.CalculatorFeature.AddTwoNumbers("Spa ces",null)"

And this is where the FQN generation comes into place. This is the FQN from the first run which gets passed as a parameter to the second run.

[OsirisTerje]

Thanks! This would certainly trigger the FQN issue.

[fernandozpiccin]

Is there any fix/workaround for this yet?

[rsuk-mb]

Thanks! This would certainly trigger the FQN issue.

@OsirisTerje Do you think the FQN issue is likely to be actioned at this point? Trying to identify the prospect of a workaround or looming solution as this is preventing us using the retry functionality in DevOps currently as many of our data driven test cases contain problematic characters (e.g. #).

[OsirisTerje]

@rsuk-mb We're still waiting .... I hope it will be within this year though.

FWIW: I would sanitize the test cases - it should not be too much extra work. You can derive a class from TestCaseData and use that.

[akshaygit09]

Is there any fix/workaround for this yet?

[OsirisTerje]

@akshaygit09 Unfortunately no, we haven't heard anything about this for a long time.