Feature: Fail when no scenarios found

[sashokbg]

Feature: Fail when no scenarios were found

  As a test automation developer
  I would like cucumber js to fail when it does not detect any scenarios
  To make sure that cucumber-js does not falsely report that everything is ok

Current state:

Example: Bad configuration
  Given I have badly configured my cucumber-js
  When I run my tests
  Then Cucumber will not find any test cases
  And Exit with code 0

Desired functionality:

Example: Bad configuration exits with error
  Given I have badly configured my cucumber-js
  When I run my tests from CLI with a special flag
  Then Cucumber will not find any tests cases
  But it will exit with error code 1

There is a discussion in slack and some people proposed --minimum-tests 1 or --expect-tests

[luke-hill]

As mentioned in slack, this is "likely" something that would fit into the "not-cucumber" category.

If this was something you wanted to make as an off-shoot plugin, then go ahead. But I'm not really sure of the need in a ubiquitous spec for cucumber.

If cucumber-js wanted to deviate and maintain this, then that's up to the maintainers here. Primarily @davidjgoss - but it's not something that would be welcome across all flavours of cucumber

[sashokbg]

I have just noticed that Jest actually implements this feature:

When having no tests found it fails with error code 1 and optionally can have a flag that disables this.

Actually when we think about it having 0 tests is never OK. If you have no tests implemented just don't put a test framework, other ways it means something went bad and you should fail.

No tests found, exiting with code 1
Run with `--passWithNoTests` to exit with code 0

https://jestjs.io/docs/cli#--passwithnotests

[mpkorstanje]

For Cucumber JVM I generally find it prudent to follow the ecosystems conventions. For Java that comes down to treating the set of empty tests as passing by default.

cucumber/cucumber-jvm#2684

[sashokbg]

Hello @mpkorstanje I did not understand what you mean to say by "ecosystem conventions" and what Java empty sets has to do with simple validating that a test framework works.

Talk about conventions it seems a lot more logical to have a convention saying that a test's job is to validate that a system is working properly and this covers the use case illustrated in this issue.

PS: Another test framework that I am using has a proper detection of "0 tests found" - ArchUnit:

ArchUnit Log:

java.lang.AssertionError: Rule 'classes that are annotated with @te should have simple name ending with 'Entity'' failed to check any classes. This means either that no classes have been passed to the rule at all, or that no classes passed to the rule matched the that() clause. To allow rules being evaluated without checking any classes you can either use ArchRule.allowEmptyShould(true) on a single rule or set the configuration property archRule.failOnEmptyShould = false to change the behavior globally.

[luke-hill]

@sashokbg The phrase "proper" implies that cucumber reporting 0 tests is incorrect. This isn't the case. Cucumber isn't a testing framework. Furthermore, it is correct to grab 0 tests as we want to cover use cases where for instance tag filtering or folder filtering is working correctly.

Within a testing framework you could also argue about 0 being correct/incorrect. I have a cucumber framework (Which also uses rspec), and I ran rspec and it correctly returned exit code 0.

[09:16:30] luke@luke-ubuntu:~/Code/folder$ rspec
No examples found.

Finished in 0.00022 seconds (files took 0.032 seconds to load)
0 examples, 0 failures

[09:17:23] luke@luke-ubuntu:~/Code/folder$ echo $?
0

[mpkorstanje]

@sashokbg in the Java ecosystem I would consider JUnit 5, Maven and Gradle to be representative. But I don't think we need to have a discussion about Java here. Ideally, since you're advancing this argument for cucumber-js, you could provide examples from the JS ecosystem that treat the set of empty tests as a failure.

[vitalets]

Maybe just a config setting (like --pass-with-no-tests) and it becomes flexible for everybody

[luke-hill]

If this is a common custom in the majority of JS frameworks @vitalets (such as mocha / chai / jest ....) then I don't see any argument against it.

But I don't have enough knowledge of JS, if it seems as though lots of people are asking for this thing and it aligns nicely then push some work up and have one the JS contributors review it. I think by now we've resolved the "how" it would need to work, given that we've explained by and large cucumber reporting 0 scenarios ran shouldn't be considered (for now), as erroneous.