"Server disconnected" on HTTP/2 connection pool with big keepalive_expiry.
#2983
Comments
|
Thanks @jonathanslenders. First places I'd start narrowing this down are... Different HTTP versions.
Different clients.
Different servers.
|
|
Hi @jonathanslenders |
|
Thank you @tomchristie for looking into it! I've been diving deeper. Maybe I have a fix (see below). I think the problem is due to the complexity of terminating TLS connections. It's hard to half-close a TLS connection, and by doing so, the underlying TCP stream is not necessarily closed, which makes it very hard for the other end to observe a half closed TLS connection. From httpx's point of view, when performing the second request, the Our From Hypercorn's point of view, I can see that Hypercorn does call Diving into httpcore, and looking at The following httpcore patch does detect whether the connection is alive. I don't know which exception to raise, but this could be a starting point: diff --git a/httpcore/_async/http2.py b/httpcore/_async/http2.py
index 8dc776f..e08e018 100644
--- a/httpcore/_async/http2.py
+++ b/httpcore/_async/http2.py
@@ -136,6 +136,10 @@ class AsyncHTTP2Connection(AsyncConnectionInterface):
self._request_count -= 1
raise ConnectionNotAvailable()
+ alive = await self._network_stream.is_alive()
+ if not alive:
+ raise Exception('Not alive!')
+
try:
kwargs = {"request": request, "stream_id": stream_id}
async with Trace("send_request_headers", logger, request, kwargs):
diff --git a/httpcore/_backends/anyio.py b/httpcore/_backends/anyio.py
index 1ed5228..d18217f 100644
--- a/httpcore/_backends/anyio.py
+++ b/httpcore/_backends/anyio.py
@@ -1,5 +1,6 @@
import ssl
import typing
+from collections import deque
import anyio
@@ -19,10 +20,13 @@ from .base import SOCKET_OPTION, AsyncNetworkBackend, AsyncNetworkStream
class AnyIOStream(AsyncNetworkStream):
def __init__(self, stream: anyio.abc.ByteStream) -> None:
self._stream = stream
+ self._buffer = deque()
async def read(
self, max_bytes: int, timeout: typing.Optional[float] = None
) -> bytes:
+ if len(self._buffer) > 0:
+ return self._buffer.popleft()
exc_map = {
TimeoutError: ReadTimeout,
anyio.BrokenResourceError: ReadError,
@@ -92,6 +96,25 @@ class AnyIOStream(AsyncNetworkStream):
return is_socket_readable(sock)
return None
+ async def is_alive(self) -> bool:
+ """
+ Test whether the connection is still alive. If this is a TLS stream,
+ then that is different from testing whether the socket is still alive.
+ It's possible that the other end did a half-close of the TLS stream,
+ but not a half-close of the underlying TCP stream.
+
+ We test whether it's alive by trying to read data. When we get b"", we
+ know EOF was reached.
+ """
+ try:
+ data = await self.read(max_bytes=1, timeout=0.01)
+ except ReadTimeout:
+ return True
+ if data == b'':
+ return False
+ self._buffer.append(data)
+ return True
+
class AnyIOBackend(AsyncNetworkBackend):
async def connect_tcp( |
|
Looks similar to the previously resolved encode/httpcore#679 Thanks for all the detailed commentary! |
When httpx is used to connect to an http/2 server, and the server disconnects, then httpx will attempt to reuse this connection for the following request. It happens when
keepalive_expiryin the httpx limits is bigger than thekeep_alive_timeoutof the HTTP/2 web server.To reproduce, we can run a Hypercorn http/2 server using a dummy Starlette application like this (TLS is required because of http/2):
Then run this client code:
Important is to ensure
http2=Trueis set andkeepalive_expiryisNoneor a value bigger than the corresponding option of the Hypercorn server.For the second request, httpx tries to reuse the connection from the connection pool although it has been closed by the other side in the meantime and fails, immediately producing the following
Server disconnectederror:I started by commenting in this discussion: #2056 , but turned it in an issue after I had a clear reproducible case. Feel free to close if it's not appropriate.
The text was updated successfully, but these errors were encountered: