Connection pool poisoning #949
Unanswered
Booplicate
asked this question in
Potential Issue
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
python 3.11.5
httpcore 1.0.5
httpx 0.26.0
Note, timeouts and limits were left default.
After stress testing, a service lost ability to send requests using "global"
AsyncClient
. At some point of testing the service started gettingPoolTimeout
exceptions, which is to be expected with the used default limits. However, even after a few hours of letting the service to "cool down" it still was gettingPoolTimeout
, but the service was nearly idle sending at most ~3 r/s. When I checked the logs for tracebacks, I found an interesting line:<AsyncConnectionPool [Requests: 0 active, 1 queued | Connections: 100 active, 0 idle]>
. The pool had 100 "active" connections and 1 queued request every time the service was trying to send a request. Active connections never dropped below 100 during those 2 hours. It was only fixed after a restart.I haven't found anything suspicious in httpcore code. I tried to create a small reproducible example, however in that case pool was getting cleared.
There's many factors, I'm not sure I'd be able to find the root of the problem. Like getting HTTP errors before testing began, testing was higher load that usual (never reached the point of
PoolTimeout
s before), HTTP version could play the role. Leaving this open in case somebody else would face a similar issue. I might try to debug this again later.Beta Was this translation helpful? Give feedback.
All reactions