[Bug]: High concurrent number of requests causes RecursionError

[Yazan-Sharaya]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When a high number of concurrent requests are sent using pymilvus, a RecursionError is raised.
This seems to happen when issuing more than 490 requests at the same time.
The recursion error traceback is very long and contains a lot of duplicates, I tried to only include the
unique/interesting parts.

  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        continuation, client_call_details, request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        continuation, client_call_details, request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        continuation, client_call_details, request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        continuation, client_call_details, request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        continuation, client_call_details, request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        continuation, client_call_details, request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/pymilvus/client/async_interceptor.py", line 45, in intercept_unary_unary
    return await continuation(new_details, new_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 685, in _run_interceptor
    return UnaryUnaryCall(
        request,
    ...<8 lines>...
        self._loop,
    )
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_call.py", line 565, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        channel.call(method, deadline, credentials, wait_for_ready),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        loop,
        ^^^^^
    )
    ^
RecursionError: maximum recursion depth exceeded
.....
Traceback (most recent call last):
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 320, in __del__
  File "/home/yazan/Desktop/dsyche/.venv/lib/python3.13/site-packages/grpc/aio/_interceptor.py", line 362, in cancel
RecursionError: maximum recursion depth exceeded
Error in sys.excepthook:

Original exception was:

Expected Behavior

I except that no error would be raised, or at least a useful error that says the connection limit is exhausted and just
reject new connections.
I've scoured the docs for any mention of an upper limit for concurrent connections (or rather requests) but couldn't find anything ):
I also looked at milvus.yaml configuration file and there seems to be some contenders that could fix the issue, but the config is quite long and there are a lot of options to tweak.

Steps/Code To Reproduce behavior

n_requests = 500
tasks = []
for i in range(n_requests):
  tasks.append(asyncio.create_task(async_engine.query("my_collection", limit=1)))
await asyncio.gather(*tasks)

Setting n_requests doesn't cause any problems and all the requests are processed very quickly!

Adding a small sleep between requests (jitter), seems to fix the issue:

import random
async def make_request():
  await asyncio.sleep(random.uniform(0, 0.1)
  await async_engine.query("my_collection", limit=1)

But the problem with this is that the amount of jitter needs to be adjusted for the request count.
For example, this works for 500 requests, however it doesn't for 10,000.
For 10,000, I had to use a sleep amount between 0.1 and 0.5 (random.uniform(0.1, 0.5)).

Environment details

- Hardware/Softward conditions (OS, CPU, GPU, Memory): Linux 6.17, Zen+ Ryzen, 16 GiB 
- Method of installation (Docker, or from source): Docker compose installation
- Milvus version (v0.3.1, or v0.4.0):  v2.5.3
- Milvus configuration (Settings you made in `server_config.yaml`): The default.