Skip to content

Fixing flaky test #1081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fixing flaky test #1081

wants to merge 3 commits into from

Conversation

ndeneh93
Copy link

@ndeneh93 ndeneh93 commented Sep 7, 2025

What is the purpose of this PR

  1. This PR fixes the error resulting from two flaky tests: org.jboss.as.cli.impl.aesh.HelpSupportTestCase and org.jboss.as.cli.impl.BootScriptInvokerTestCase
  2. The mentioned tests may fail or pass without changes made to the source code when it is run in different JVMs due to HashSet's non-deterministic iteration order.

Why the tests fail

  1. Test one fails because HelpSupportTestCase calls SynopsisGenerator, which uses a HashSet as its data structure for the conflicts object. As per Java 11 documentation, the ordering of HashSet is not constant. So, the test may fail as the given order can be different from the expected order.
  2. Test two fails because BootScriptInvokerTestCase#test uses a HashSet as its data structure for the echos object. As per Java 11 documentation, the ordering of HashSet is not constant. So, the test may fail as the given order can be different from the expected order.

How to reproduce the test failure

  1. Run the tests with NonDex maven plugin. The commands to recreate the flaky test failures are:
  2. mvn -pl cli edu.illinois:nondex-maven-plugin:2.1.1:nondex
  3. Dtest=org.jboss.as.cli.impl.aesh.HelpSupportTestCase#testStandalone
  4. mvn -pl cli edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.jboss.as.cli.impl.BootScriptInvoker#test
  5. Fixing the flaky test now may prevent flaky test failures in future Java versions.

Expected Results
Both tests should run successfully when run with NonDex.

Actual Results
I got the following test error: Execution default-cli of goal edu.illinois:nondex-maven-plugin:2.1.1:nondex failed: Unsupported class file major version 65 TIME elapsed: Total time: 18.502 s, Finished at: 2025-09-07T18:12:11Z,

Description fix:
For test one: Changed the object type of conflicts2 from HashSet to TreeSet in method org.jboss.as.cli.impl.aesh.SynopsisGenerator#addSynopsisOption(SynopsisOption).
For test two: Changed the object type of echos from HashSet to LinkedHashSet in org.jboss.as.cli.impl.BootScriptInvokerTestCase#test

The Jira Issue
https://issues.redhat.com/browse/WFCORE-6214

@ndeneh93 ndeneh93 marked this pull request as ready for review September 7, 2025 21:41
@ndeneh93 ndeneh93 closed this Sep 11, 2025
@ndeneh93 ndeneh93 reopened this Sep 11, 2025
@ndeneh93
name: Flaky Test Check
d9752c9 
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up JDK 17
      uses: actions/setup-java@v3
      with:
        java-version: '17'
    - name: Install Maven
      run: sudo apt-get install maven -y
    - name: Run flaky test with NonDex
      run: mvn edu.illinois:nondex-maven-plugin:nondex -Dtest=typescript-generator-core.MyFlakyTest#shouldGenerateCode
@ndeneh93
Add NonDex workflow for flaky test
a3729f3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ndeneh93