Intermittent httpcore.RemoteProtocolError: Server disconnected without sending a response. #2056

jogu · 2022-01-31T19:09:28Z

jogu
Jan 31, 2022

Hi

I've been trying to switch from requests to httpx for all kinds of reasons, but I'm getting a weird and kind of reproducible problem.

About 1 in 5000 or so requests I'll get:

httpcore.RemoteProtocolError: Server disconnected without sending a response.

I'm running the latest httpx on the latest python:3.10-alpine3.15 docker container.

"By kind of reproducible" I mean "generally happens upto 2 or 3 times inside a 25 minute job in our CI".

This happens with both the normal and async client.

The client is talking to apache (as a reverse proxy for a java spring app) on a Debian container, and I've tried the apaches across 3 major versions of Debian (stretch, buster, bullseye) and the problem happens on all 3.

I've had some success adding a custom transport that retries in this case, though I'd prefer to try and fix the root cause if possible as I don't think this should happen.

I can't figure out how to diagnose this further. I tried HTTPX_LOG_LEVEL=trace, but as noted in other discussions that's not really doing much currently.

The backtrace looks like this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 244, in handle_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 228, in handle_request
    response = connection.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 90, in handle_request
    return self._connection.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 102, in handle_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 81, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/builds/openid/conformance-suite-private/../conformance-suite/scripts/run-test-plan.py", line 285, in run_test_module
    test_module_info = await conformance.create_test_from_plan_with_variant(plan_id, module, moduledict.get('variant'))
  File "/builds/openid/conformance-suite/scripts/conformance.py", line 131, in create_test_from_plan_with_variant
    response = self.requests_session.post(api_url, params=payload)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1116, in post
    return self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 802, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 889, in send
    response = self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 917, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 990, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

johtso · 2022-03-30T14:49:52Z

johtso
Mar 30, 2022

I'm having the same issue with about 1 in 10 requests..

4 replies

tomchristie Mar 30, 2022
Maintainer

Okay, so I can tell ya (almost certainly) why this occurs, but what we do need is to narrow down a bit more how we can get it to reproduce, and see if there's anything we can do to have more resilience to it.

Here's the deal...

Make a request to a server, get a response.
The connection returns to the connection pool.
If the connection hasn't expired, and hasn't been closed by the server then we can re-use the connection on the next request.
Except that sometimes the connection has actually been dropped by the server, without the connection having been properly closed. In which case we successfully send the request over the connection, and only "see" the connection as having been dropped when we attempt to read the response.

I'd really like to get a nice simple example of how we can reliably replicate this, so we can have a proper dig into if our connection-has-expired logic is meaningfully different from urllib3 or aiohttp, and figure out if there's a more robust way we can check for connection aliveness when re-acquiring connections from the pool.

tomchristie Mar 30, 2022
Maintainer

Our is-connection-alive check...

https://github.com/encode/httpcore/blob/44b8eedfe3bc3c4b15d54810c36f05e7c5fd600d/httpcore/_utils.py#L7-L36

Which we're copying from trio...

https://github.com/python-trio/trio/blob/2f81928babefcb55915a5ec9042c764121e92a0d/trio/_socket.py#L471-L478

What urllib3 use...

https://github.com/urllib3/urllib3/blob/fab33936db9281e96b6325b4fafe2fbb852fa458/src/urllib3/util/connection.py#L11-L22

I can't figure out what aiohttp use for determining connection aliveness. Possibly structured differently because they're using asyncio protocols, rather than a Stream API. (?) ... https://github.com/aio-libs/aiohttp/blob/936e682d1ab6c833b3e5f0cc3596882cb9cb2444/aiohttp/client_proto.py#L75

tomchristie Mar 30, 2022
Maintainer

@johtso I'd be interested to know what you get from the following on your system...

import select

p = select.poll()
p.poll(0)

johtso Mar 30, 2022

@tomchristie

Python 3.10.2 (main, Feb  2 2022, 05:51:25) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import select
>>>
>>> p = select.poll()
>>> p.poll(0)
[]
>>>

tomchristie · 2022-03-30T16:45:03Z

tomchristie
Mar 30, 2022
Maintainer

@jogu - Could you start by updating httpx and httpcore to the lastest versions? (And then show the traceback once you've done that)

For example looking at this line in the traceback...

  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 228, in handle_request
    response = connection.handle_request(request)

I can see that doesn't match up with the latest codebase...

Line 228 is here...

https://github.com/encode/httpcore/blob/44b8eedfe3bc3c4b15d54810c36f05e7c5fd600d/httpcore/_sync/connection_pool.py#L228

Whereas the connection.handle_request line is actually on line 237...

https://github.com/encode/httpcore/blob/44b8eedfe3bc3c4b15d54810c36f05e7c5fd600d/httpcore/_sync/connection_pool.py#L237

3 replies

jogu Mar 30, 2022
Author

@tomchristie I should probably add an update to this ticket. As I was never able to get any useful diagnostics out of httpx I never discovered exactly what was going wrong, but the problem seems like it was maybe related to timeouts, either on the httpx side or the server side (we had one spring hosted API that sometimes failed to complete before spring's timeout hit). Either way once I increased both sets of timeout (i.e. self.httpclient = httpx.Client(verify=verify_ssl, transport=transport, timeout=20) and increasing the spring timeout so it didn't give up either) things started behaving more sensibly. It also turned out I wasn't calling httpclient.close() when something went wrong, which may have somehow contributed (httpcore 0.14.6 made it obvious that I'd got that wrong).

I could have another go at reproducing if it's helpful, let me know if johtso's example script doesn't help.

tomchristie Mar 31, 2022
Maintainer

Thanks. So based on looking into this with the reproduction shown by @johtso, I think you'd likely see exactly the same behaviour with either requests or httpx or aiohttp here.

We can't guard against this completely. For example as seen in @johtso's case, some server setups just outright fail to handle keep-alive connections reliably. However I think we could do a much better job of being resilient to these cases...

We could default to limiting the maximum overall time that we keep connections open for. For example, don't re-use a connection if it's been alive for more than 5 minutes.
We could default to limiting the maximum number of requests that connection will service before treating it as expired. For example, don't re-use a connection if it's sent more than 50 requests.
We could improve our error messaging in this case. Possibly including more information about the connection would be helpful. https://github.com/encode/httpcore/blob/master/httpcore/_sync/http11.py#L189-L191
We could also consider automatic retries for idempotent HTTP methods in this case...

See RFC 2616 Sec 8.1.4...

A client, server, or proxy MAY close the transport connection at any time. For example, a client might have started to send a new request at the same time that the server has decided to close the "idle" connection. From the server's point of view, the connection is being closed while it was idle, but from the client's point of view, a request is in progress.

This means that clients, servers, and proxies MUST be able to recover from asynchronous close events. Client software SHOULD reopen the transport connection and retransmit the aborted sequence of requests without user interaction so long as the request sequence is idempotent (see section 9.1.2). Non-idempotent methods or sequences MUST NOT be automatically retried, although user agents MAY offer a human operator the choice of retrying the request(s). Confirmation by user-agent software with semantic understanding of the application MAY substitute for user confirmation. The automatic retry SHOULD NOT be repeated if the second sequence of requests fails.

jogu Mar 31, 2022
Author

We could also consider automatic retries for idempotent HTTP methods in this case...

I'd say this is probably the most important one to go for.

It would be great if the application got some say as to what is/isn't idempotent too; there are quite a few cases where it's basically okay to retry a POST.

johtso · 2022-03-30T16:48:13Z

johtso
Mar 30, 2022

@tomchristie I've made a fairly minimal reproduction:

import httpx

urls = """\
https://archive.org/download/cu31924003160540/page/n0_x0_y0_w1283_h2099_s2.jpg
https://archive.org/download/cu31924003160540/page/n1_x532_y904_w220_h222_s1.jpg
https://archive.org/download/cu31924003160540/page/n2_x404_y1804_w326_h92_s1.jpg
https://archive.org/download/cu31924003160540/page/n3_x130_y270_w1024_h368_s2.jpg
https://archive.org/download/cu31924003160540/page/n7_x54_y214_w1184_h1342_s2.jpg
https://archive.org/download/cu31924003160540/page/n39_x174_y450_w902_h904_s2.jpg
https://archive.org/download/cu31924003160540/page/n50_x172_y452_w958_h968_s2.jpg
https://archive.org/download/cu31924003160540/page/n55_x210_y424_w450_h214_s1.jpg
https://archive.org/download/cu31924003160540/page/n55_x572_y628_w526_h238_s2.jpg
https://archive.org/download/cu31924003160540/page/n56_x220_y488_w872_h856_s2.jpg
https://archive.org/download/cu31924003160540/page/n58_x232_y730_w844_h344_s2.jpg
https://archive.org/download/cu31924003160540/page/n59_x168_y740_w966_h390_s2.jpg
https://archive.org/download/cu31924003160540/page/n60_x176_y868_w994_h458_s2.jpg
https://archive.org/download/cu31924003160540/page/n66_x236_y682_w802_h252_s2.jpg
https://archive.org/download/cu31924003160540/page/n80_x368_y672_w618_h528_s2.jpg
https://archive.org/download/cu31924003160540/page/n88_x152_y292_w1020_h1298_s2.jpg
https://archive.org/download/cu31924003160540/page/n92_x304_y116_w708_h1702_s2.jpg
https://archive.org/download/cu31924003160540/page/n98_x870_y412_w284_h958_s1.jpg
https://archive.org/download/cu31924003160540/page/n103_x260_y570_w776_h814_s2.jpg
https://archive.org/download/cu31924003160540/page/n119_x210_y592_w884_h536_s2.jpg
https://archive.org/download/cu31924003160540/page/n120_x562_y966_w586_h604_s2.jpg
https://archive.org/download/cu31924003160540/page/n147_x140_y342_w894_h1266_s2.jpg
https://archive.org/download/cu31924003160540/page/n152_x504_y498_w646_h586_s2.jpg
""".split("\n")

async def get_image(url, client):
    print('fetching', url)
    response = await client.get(url, follow_redirects=True)
    print(repr(response))
    

async def gather_with_concurrency(n, *tasks):
    semaphore = asyncio.Semaphore(n)

    async def sem_task(task):
        async with semaphore:
            return await task

    return await asyncio.gather(*(sem_task(task) for task in tasks))


async def main():
    client = httpx.AsyncClient(timeout=20.0)
    await gather_with_concurrency(2, *[get_image(url, client) for url in urls])


if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

6 replies

johtso Mar 31, 2022

I think the follow_redirects=True is needed? I don't get the issue if we don't follow the redirect.

Ah the simplified example makes sense, it's the fact that a connection pool is being use that triggers the issue, nothing to do with async (I wasn't using the client api when I tried it non-async).

tomchristie Mar 31, 2022
Maintainer

I think the follow_redirects=True is needed? I don't get the issue if we don't follow the redirect.

Yes correct. I believe the application server which handles requests on archive.org:443 is operating just fine, but the media servers which handle requests on domains such as ia800209.us.archive.org:443 are not consistently dealing with keep-alive connections properly.

You can resolve the issue for your case by switching off keep-alive...

with httpx.Client(headers={"Connection": "close"}) as client:
    ...

johtso Mar 31, 2022

@tomchristie awesome! was about to ask if there was a way to switch off connection pooling while still using the Client api, will give that a go.

jogu Mar 31, 2022
Author

Yes correct. I believe the application server which handles requests on archive.org:443 is operating just fine, but the media servers which handle requests on domains such as ia800209.us.archive.org:443 are not consistently dealing with keep-alive connections properly.

I think they are actually very consistently dealing with keep-alive :-) They seem to be running a relatively modern nginx. The advantage of these servers is they are exposed on http too (i.e. no TLS) so it's easy to Wireshark and see exactly what's going on, which I just did.

They appear to have a pretty aggressive idle timeout - I make it to be around 800ms. I think that's what's causing the issue.

adnaanbheda Apr 6, 2022

I've also been facing this issue but the magnitude at which this happened was much larger, around 300-400 requests died out of 50K requests. The server I'm accessing is running uvicorn, so my first solution was to increase the keep-alive config in uvicorn significantly, but that didn't alleviate the problem, I finally resorted to disable connection pooling i.e. recreating the client every request, that dipped the number significantly to about 2-3 requests out of 50K.

The thing that I've noticed is that the error has a turnaround time of around 5-10ms in my case, so maybe a retry mechanism can help solve the issue.

jonathanslenders · 2023-12-05T15:07:02Z

jonathanslenders
Dec 5, 2023

I'm facing the same issue (I think it's the same). It happens when talking to a Hypercorn ASGI server over HTTP/2. It doesn't happen always but very regularly. A few requests is all that's needed to reproduce it.
The main problem is that once the "Server disconnected" happens, the connection pool seems to become unusable and every following request will fail with the same error.

This is the traceback:

  Traceback (most recent call last):
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
      yield
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
      resp = await self._pool.handle_async_request(req)
    File ".../lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
      raise exc
    File ".../lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
      response = await connection.handle_async_request(request)
    File ".../lib/python3.9/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
      return await self._connection.handle_async_request(request)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 185, in handle_async_request
      raise exc
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 148, in handle_async_request
      status, headers = await self._receive_response(
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 292, in _receive_response
      event = await self._receive_stream_event(request, stream_id)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 333, in _receive_stream_event
      await self._receive_events(request, stream_id)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 361, in _receive_events
      events = await self._read_incoming_data(request)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 452, in _read_incoming_data
      raise exc
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 440, in _read_incoming_data
      raise RemoteProtocolError("Server disconnected")
  httpcore.RemoteProtocolError: Server disconnected

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    ...
    File ".../our-code.py", line 191, in make_stream_request
      async with client.stream(
    File "/home/jonathan/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 181, in __aenter__
      return await self.gen.__anext__()
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream
      response = await self.send(
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
      response = await self._send_handling_auth(
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
      response = await self._send_handling_redirects(
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
      response = await self._send_single_request(request)
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
      response = await transport.handle_async_request(request)
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
      resp = await self._pool.handle_async_request(req)
    File "/home/jonathan/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 137, in __exit__
      self.gen.throw(typ, value, traceback)
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
      raise mapped_exc(message) from exc
  httpx.RemoteProtocolError: Server disconnected

The client is created like this:

async with httpx.AsyncClient(
                  verify=ssl_context,
                  timeout=10.0,
                  trust_env=False,
                  proxies=None,
                  http2=True,
                  limits=httpx.Limits(
                      max_connections=100,
                      max_keepalive_connections=20,
                      keepalive_expiry=None,
                  ),
              ) as client: ...

edit: I think, but really not sure at all, that it happens after a timeout, possibly faster with max_keepalive_connections=1. It doesn't seem to happen if I keep firing requests in a loop.

1 reply

jonathanslenders Dec 6, 2023

I created this issue with a simple example to replicate it: #2983

facundopadilla · 2024-08-25T18:08:44Z

facundopadilla
Aug 25, 2024

Hi guys, I had the same issue, but, I found some things related about this:

If you use a VPN, try turn off it
It's possible that you IP is banned/black listed, this happened to me.
The server is off

I was use tenacity for retry and sometimes fixed the problem, but it come back likewise 🙃

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent httpcore.RemoteProtocolError: Server disconnected without sending a response. #2056

{{title}}

Replies: 5 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Intermittent httpcore.RemoteProtocolError: Server disconnected without sending a response. #2056

Replies: 5 comments · 14 replies

tomchristie Mar 30, 2022 Maintainer

tomchristie Mar 30, 2022 Maintainer

tomchristie Mar 30, 2022 Maintainer

tomchristie Mar 30, 2022 Maintainer

jogu Mar 30, 2022 Author

tomchristie Mar 31, 2022 Maintainer

jogu Mar 31, 2022 Author

tomchristie Mar 31, 2022 Maintainer

jogu Mar 31, 2022 Author

Replies: 5 comments 14 replies

tomchristie Mar 30, 2022
Maintainer

tomchristie Mar 30, 2022
Maintainer

tomchristie Mar 30, 2022
Maintainer

tomchristie
Mar 30, 2022
Maintainer

jogu Mar 30, 2022
Author

tomchristie Mar 31, 2022
Maintainer

jogu Mar 31, 2022
Author

tomchristie Mar 31, 2022
Maintainer

jogu Mar 31, 2022
Author