Non deterministic tests in the repository

[yugvajani]

Expected Behavior

The repository contains flaky tests that behave inconsistently—passing at times and failing at others for the same code. Ideally, these tests should always pass. To identify the flaky tests, I used the NonDex tool. NonDex operates by systematically randomizing the iteration order of standard Java library classes (e.g., HashSet, HashMap) during test execution. By altering the order of elements each time tests are run, NonDex helps reveal potential bugs caused by dependencies on specific ordering, which could lead to flaky or unreliable tests. Tests that are not flaky should run successfully with this tool, as expected.

Current Behavior

The repository contains flaky tests that occasionally fail when run with the NonDex tool.

Possible Solution

The primary cause of this flakiness is that assertEquals enforces order dependency when comparing strings. Instead, if we directly compare the JSON objects or ignore the order while checking for equality, the tests will no longer be flaky.

Steps to Reproduce (for Bugs)

Consider one of the test which i found org.openhab.core.cache.ExpiringCacheMapTest.testValues:

1. Go to the root of the repository.
2. Build the module which has the flaky test:
   mvn install -pl bundles/org.openhab.core -am -DskipTests

The generalized format of command is :
mvn install -pl <module-name> -am -DskipTests

3. Then, run the regular test to verify that it is not a failing test but a flaky:
   mvn -pl bundles/org.openhab.core test -Dtest=org.openhab.core.cache.ExpiringCacheMapTest#testValues

The generalized format of command is :
mvn -pl <module-name> test -Dtest=<test-path>

4. To identify the flaky test, execute the following nondex command:
   mvn -pl bundles/org.openhab.core edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=org.openhab.core.cache.ExpiringCacheMapTest#testValues

The generalized format of command is :
mvn -pl <module-name> [test](edu.illinois:nondex-maven-plugin:2.1.7:nondex) -Dtest=<test-path>

Found 26 flaky tests using this tool in the repository.

Your Environment

• Environment name and version : Java 17, Maven 3.9
• Operating System and version : Ubuntu