[SUREFIRE-1963] Detecting multiple test-frameworks

[slawekjaranowski]

Following this checklist to help us incorporate your
contribution quickly and easily:

• Make sure there is a JIRA issue filed
  for the change (usually before you start working on it). Trivial changes like typos do not
  require a JIRA issue. Your pull request should address just this issue, without
  pulling in other changes.
• Each commit in the pull request should have a meaningful subject line and body.
• Format the pull request title like [SUREFIRE-XXX] - Fixes bug in ApproximateQuantiles,
  where you replace SUREFIRE-XXX with the appropriate JIRA issue. Best practice
  is to use the JIRA issue title in the pull request title and in the first line of the
  commit message.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Run mvn clean install to make sure basic checks pass. A more thorough check will
  be performed on your pull request automatically.
• You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

• In any other case, please file an Apache Individual Contributor License Agreement.

[rmannibucau]

Think it must not be integrated but instead we should have the providers to be greedy, ie while we find tests with a provider they must be executed once (the first provider in the list supporting the test is used). No reason to fail when we know how to do. If user wants to enforce a particular engine then he must configure it but it is the design of surefire already so failling wouldn't help IMHO.

[slawekjaranowski]

Make providers to be greedy it is not easy, we have many corrner cases.

The most of projects (99%) not need support for multiple providers and one provider it is ok,
or must be configured manually if want to use multiple framework.

We shouldn't complicate code for detecting and executing provaiders only for a few cases - it is complicated enough now.

This change don't change current behavior, with empty multipleFrameworks nothing change
but add possibility to detecting wrong test configuration.

[Tibor17]

@rmannibucau for you information @slawekjaranowski has one situation in his work which is related to human factor when some colleagues update the dependencies. The QA team wants to be notified when such situation happens and then the QA team would add a new provider. This way no test would be isolated from execution. Basically this is my understanding so far.
Maybe some generic approach should be found useful for many users and for Slawomir too. Currently the resolution of providers is hardcoded but we can open the api and the user may change the logic without making changes in the ASF project.

[slawekjaranowski]

The problem which I have (and other users - look in linked issues) is that in some of situation when dependency tree of project can change because of transitive dependency and so on... another testing library/ framework get be available on project - and in result is possibility that some test will be skipped silently.

I think that get possibility to detect such situation is enough, in standard case one project use one testing framework ... so I will not complicate more code of surefire for such corner case.

[Tibor17]

@slawekjaranowski
It would be great to have a sample project, in your GH, with several options and the playground with dependencies and the ignored tests. I think, we should start this way and we do it in customers the same way when the users decide to open a ticket in ASF Jira.

[slawekjaranowski]

@Tibor17
There is IT test - surefire-its/src/test/java/org/apache/maven/surefire/its/jiras/Surefire1963MultipleFrameworkIT.java
Is it not enough?