Add improvements for opensearch engine and updated the docker compose.

[navneet1v]

Description

Added some indexing and query improvements for Opensearch Engine.

Improvements include:

1. Fixed the shards and the refresh interval.
2. Corrected the force merge API.
3. Updated the engine to Faiss and ensure in case of cosine similarity IP is used with normalized vectors.
4. Added the support for dynamic index_thread_qty to speed up the indexing and force merge.

Next steps

1. Add the disk based vector search support for Opensearch in the benchmarking tool

Tests

Validated the run via the below command

python3 -m run --engines opensearch-default --datasets glove-25-angular

Testing setup:

1. VCPU : 16
2. RAM: 25GB
3. number of nodes: 1
4. Opensearch running via docker compose file present in engine/servers/opensearch-single-node

Upload results:

{
  "params": {
    "experiment": "opensearch-default",
    "engine": "opensearch",
    "dataset": "glove-25-angular",
    "parallel": 16,
    "method": {
      "parameters": {
        "m": 16,
        "ef_construction": 128
      }
    }
  },
  "results": {
    "post_upload": {},
    "upload_time": 51.88272933400003,
    "total_time": 144.80344308400527
  }
}

Search Results

{
  "params": {
    "dataset": "glove-25-angular",
    "experiment": "opensearch-default",
    "engine": "opensearch",
    "parallel": 100,
    "config": {
      "ef_search": 256
    }
  },
  "results": {
    "total_time": 14.72621393299778,
    "mean_time": 0.009195335887654801,
    "mean_precisions": 0.9821270000000001,
    "std_time": 0.00690122954616666,
    "min_time": 0.0020300589967519045,
    "max_time": 0.07892615199671127,
    "rps": 679.0611657211151,
    "p95_time": 0.022183901458629394,
    "p99_time": 0.03174201674788492
  }
}

{
  "params": {
    "dataset": "glove-25-angular",
    "experiment": "opensearch-default",
    "engine": "opensearch",
    "parallel": 1,
    "config": {
      "ef_search": 512
    }
  },
  "results": {
    "total_time": 26.042633678996935,
    "mean_time": 0.002513192814134527,
    "mean_precisions": 0.9955579999999999,
    "std_time": 0.00023401047721296358,
    "min_time": 0.0018990870157722384,
    "max_time": 0.013257409998914227,
    "rps": 383.98574135245303,
    "p95_time": 0.0028366548067424446,
    "p99_time": 0.003023336109181401
  }
}