[BUG] Unstable BWC CI action for 2.x branch

[martin-gaievski]

What is the bug?

It's often one or several BWC tests will fail in CI when PR is created for 2.x codeline. If same tests re-ran again that will most likely pass. That adding lots of headache when you work with PR or plugin got built in distribution CI.

We do have 120+ bwc tests for 2.x codeline due to requirements of being backward compatible with all the versions from same major release. Error can be due to racing conditions on the test runner, or maybe some corner case in the test setup logic that with high number of runs will appear with higher probability.

How can one reproduce the bug?

Submit a PR for 2.x branch. For sample runs you can check some of the recent PRs:
https://github.com/opensearch-project/neural-search/actions/runs/11708689000/job/32611076434?pr=976

Do you have any additional context?

With main/3.x PRs it almost never happens, my guess its due to less combinations of versions - for 3.x we need to run only one latest 2.x.

[dreyarlaxus]

can you assign this issue for me??

[martin-gaievski]

@dreyarlaxus sure, feel free to work on this one. Let me know when you're ready with PR and reviews

[martin-gaievski]

@dreyarlaxus you can start checking current 2.x, we have merged PR #983, so now 2.x pointing to the relevant branches