Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Applying [Explicit] and [Ignore] Attributes to Test Class with Custom Category Ignores All Tests in Category #1197

Open
dhrumil0902 opened this issue Jun 24, 2024 · 10 comments
Labels

Comments

@dhrumil0902
Copy link

dhrumil0902 commented Jun 24, 2024

Description
When applying the [Explicit] attribute and [Ignore(...)] attribute to a test class that has a custom category attribute, all classes within that category are ignored when running dotnet test while filtering by that category. Therefore no tests are ran. Please let me know if I am doing something wrong.

Steps to Reproduce

  1. Use the example code below, or create similar classes with those attributes.
  2. Run dotnet test --filter TestCategory=MyCustomCategory on the test project.

Example Code

using NUnit.Framework;

namespace TestProject1
{
    [TestFixture, Category("MyCustomCategory")]
    [Explicit]
    [Ignore("Ignoring")]
    public class MyTestClass
    {
        [Test]
        public void MyTest()
        {
            Assert.Pass();
        }
    }

    [TestFixture, Category("MyCustomCategory")]
    [Explicit]
    public class AnotherTestClass
    {
        [Test]
        public void AnotherTest()
        {
            Assert.Pass();
        }
    }
}

Expected Behavior
Only the specific test class with the [Ignore(...)] attribute should be ignored. Other test classes in the same category should execute as expected.

  Determining projects to restore...
  All projects are up-to-date for restore.
  TestProject1 -> C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll
Test run for C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.9.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Skipped MyTest [< 1 ms]

Passed!  - Failed:     0, Passed:     1, Skipped:     1, Total:     2, Duration: 39 ms - TestProject1.dll (net8.0)

Actual Behavior
All test classes within the same custom category are ignored, causing all tests in that category to be skipped.

  Determining projects to restore...
  All projects are up-to-date for restore.
  TestProject1 -> C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll
Test run for C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.9.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Skipped AnotherTest [5 ms]
  Skipped MyTest [< 1 ms]

Skipped! - Failed:     0, Passed:     0, Skipped:     1, Total:     1, Duration: 9 ms - TestProject1.dll (net8.0)

CsProj File

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.10.0" />
      <PackageReference Include="NUnit" Version="4.1.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
      <PackageReference Include="NunitXml.TestLogger " Version="3.1.20" />
  </ItemGroup>

</Project>
@dhrumil0902
Copy link
Author

dhrumil0902 commented Jun 26, 2024

@OsirisTerje pls take a look if possible =)

@OsirisTerje
Copy link
Member

OsirisTerje commented Jun 26, 2024

I just ran these in the Test explorer, and add the category as a trait,and using Run All they are not executed for different reasons. The MyTest because it is Ignored, the AnotherTest because it is Explicit, but it should have run.

You see the different icons for them:

image

If I remove the Ignore attribute, they do run both of them, as they should.

So it seems the Ignore does mess up something here. Nice catch!

If I add a third test class, with the same category, it does execute, but the two others doesnt, for the same different reasons. So, yes, this is a bug.

Repro uploaded to https://github.com/nunit/nunit.issues/tree/main/Issue4736

@manfred-brands Want to have a look ?

@dhrumil0902
Copy link
Author

@manfred-brands any updates? @OsirisTerje I can take a look at this bug if you can point me in the right direction. I also saw on your profile that you are looking for contributors. Where and how can I start contributing?

@OsirisTerje
Copy link
Member

The best way to debug this is to take the repro above, add that to a solution which also includes the framework. So clone both into their respective local repos, and let the solution span both projects.
Then it is easy to get into debugging them, and see what happens with the filter.
Note, it may be a bug in the adapter too, but currently it does look like the error is in the framework. To make sure it is in the framework and not adapter, it could be wise to run the repro using NUnitLite.

About contributing: Yes, we do want more contributors. Check out the contributing guidelines and the NUnit.org website. You should also check the NUnit governance documentation which explains the roles and project structures. Also check the docs, which may be helpful.

@dhrumil0902
Copy link
Author

The bug is not there when executing tests with NUnitLite or nunit-console, so looks like the problem is in the adapter.

@OsirisTerje
Copy link
Member

Ok, thanks! I'll move this issue to the adapter.
To debug into the adapter follow the steps outlined here: https://hermit.no/debugging-the-nunit3testadapter-take-2/
But do build a debug version, it makes it easier for going in there.

@OsirisTerje OsirisTerje transferred this issue from nunit/nunit Jul 25, 2024
@dhrumil0902
Copy link
Author

@OsirisTerje After debugging the code in the package, I have an idea of where this bug is coming from, but I have multiple questions. Is there a better medium to ask questions and/or ask for help?

@OsirisTerje
Copy link
Member

OsirisTerje commented Aug 2, 2024

You can join https://nunit.slack.com. Here is an invite link https://join.slack.com/t/nunit/shared_invite/zt-2o4x0nfoh-llr_V6GO5ZI4TfCXqjUUAw

@amaltinsky
Copy link

@OsirisTerje
Hi! Any updates on this bug? It's causing us some trouble, preventing us from using the [Ignore] attribute.

@OsirisTerje
Copy link
Member

@dhrumil0902 started doing something on this issue a few weeks back. Not sure where it is now, but he might answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants