[BUG] Flaky test: testReduceWithConcurrentSegmentSearch_whenMultipleCollectorsMatchedDocs_thenSuccessful

[chishui]

What is the bug?

testReduceWithConcurrentSegmentSearch_whenMultipleCollectorsMatchedDocs_thenSuccessful failed in gradle check: https://github.com/opensearch-project/neural-search/actions/runs/9788202904/job/27025892009?pr=769

org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests > testReduceWithConcurrentSegmentSearch_whenMultipleCollectorsMatchedDocs_thenSuccessful FAILED
    java.lang.IllegalArgumentException: cannot merge top docs because it does not have enough elements
        at __randomizedtesting.SeedInfo.seed([8305B866B5D072CE:EAE289E17B23685E]:0)
        at org.opensearch.neuralsearch.search.query.HybridQueryScoreDocsMerger.merge(HybridQueryScoreDocsMerger.java:39)
        at org.opensearch.neuralsearch.search.query.TopDocsMerger.merge(TopDocsMerger.java:52)
        at org.opensearch.neuralsearch.search.query.HybridCollectorManager.reduceCollectorResults(HybridCollectorManager.java:241)
        at org.opensearch.neuralsearch.search.query.HybridCollectorManager.lambda$reduce$0(HybridCollectorManager.java:135)
        at org.opensearch.neuralsearch.search.query.HybridCollectorManager.lambda$reduceSearchResults$7(HybridCollectorManager.java:254)
        at org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testReduceWithConcurrentSegmentSearch_whenMultipleCollectorsMatchedDocs_thenSuccessful(HybridCollectorManagerTests.java:412)
REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testReduceWithConcurrentSegmentSearch_whenMultipleCollectorsMatchedDocs_thenSuccessful" -Dtests.seed=8305B866B5D072CE -Dtests.security.manager=false -Dtests.locale=af-ZA -Dtests.timezone=Europe/Uzhgorod -Druntime.java=11

How can one reproduce the bug?

It can be reproduced locally with command

./gradlew ':test' --tests "org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testReduceWithConcurrentSegmentSearch_whenMultipleCollectorsMatchedDocs_thenSuccessful" -Dtests.seed=8305B866B5D072CE -Dtests.security.manager=false -Dtests.locale=af-ZA -Dtests.timezone=Europe/Uzhgorod

However, if remove the -Dtests.seed=8305B866B5D072CE, the test will pass.

What is the expected behavior?

The UT will pass with or without seed.

What is your host/environment?

linux/macos

Do you have any screenshots?

N/A

Do you have any additional context?

N/A

[chishui]

@martin-gaievski could you take a look?

[dblock]

[Catch All Triage w/ 1, 2, 3]