Memory leak on multiple async connections.

[T-256]

from frankie567/httpx-ws#42

To Reproduce

import asyncio

async def leak():
    async with httpx.AsyncClient(timeout=21) as client:
        async with client.stream("GET", "https://socketsbay.com/wss/v2/1/demo/") as response:
            pass

input("before")
asyncio.run(asyncio.gather(*[leak() for _ in range(100)]))
input("after")

Expected behavior

No memory leak

Configuration

• Python version: 3.7 (32 bit) on win 11 (64 bit)
• httpx version: 0.24.1

Additional context

In my run of above code memory bump from 10MB to 80MB

[zanieb]

@T-256 How are you measuring memory usage?

I get connection timeouts with the example website you've provided.

I've used tracemalloc to measure memory. Your example doesn't look like it gives a chance for garbage collection to run?

e.g.

import tracemalloc
import asyncio
import gc
import httpx

tracemalloc.start()


async def leak():
    async with httpx.AsyncClient(timeout=21) as client:
        async with client.stream("GET", "https://socketsbay.com/wss/v2/1/demo/") as response:
            pass

async def main():
    await asyncio.gather(*[leak() for _ in range(100)])

snapshot1 = tracemalloc.take_snapshot()
print("Prerun usage {}".format(tracemalloc.get_traced_memory()[0]))

try:
    asyncio.run(main())
except Exception:
    pass


print("Postrun usage {} (peak {})".format(*tracemalloc.get_traced_memory()))

gc.collect()

print("Postcollect usage {}".format(tracemalloc.get_traced_memory()[0]))

snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("[ Top 10 differences ]")
for stat in top_stats[:10]:
    print(stat)

[T-256]

I get connection timeouts with the example website you've provided.

Try with https://google.com

here is output of your example:

Prerun usage 248
Postrun usage 2183516 (peak 3648692)
Postcollect usage 471983
[ Top 10 differences ]
<frozen importlib._bootstrap_external>:525: size=98.1 KiB (+98.1 KiB), count=2457 (+2457), average=41 B
C:\python37x86\lib\abc.py:126: size=13.0 KiB (+13.0 KiB), count=116 (+116), average=115 B
C:\python37x86\lib\stringprep.py:187: size=10.1 KiB (+10.1 KiB), count=2 (+2), average=5150 B
C:\python37x86\lib\dataclasses.py:387: size=8167 B (+8167 B), count=137 (+137), average=60 B
<frozen importlib._bootstrap>:219: size=6969 B (+6969 B), count=70 (+70), average=100 B
C:\python37x86\lib\weakref.py:409: size=6304 B (+6304 B), count=2 (+2), average=3152 B
C:\python37x86\lib\stringprep.py:262: size=5748 B (+5748 B), count=98 (+98), average=59 B
E:\proj\.venv\lib\site-packages\anyio\_backends\_asyncio.py:1628: size=5712 B (+5712 B), count=12 (+12), average=476 B
C:\python37x86\lib\abc.py:143: size=5171 B (+5171 B), count=66 (+66), average=78 B
C:\python37x86\lib\_weakrefset.py:84: size=5120 B (+5120 B), count=2 (+2), average=2560 B

And here is memory usage in TaskManager at each step:

• Prerun: 11MB
• Postrun: 90MB
• Postcollect: 25MB

[T-256]

Without gc collect:

Prerun usage 248
Postrun usage 2388886 (peak 3723259)
[ Top 10 differences ]
C:\python37x86\lib\asyncio\locks.py:244: size=139 KiB (+139 KiB), count=1363 (+1363), average=104 B
E:\proj\.venv\lib\site-packages\h11\_util.py:92: size=118 KiB (+118 KiB), count=2748 (+2748), average=44 B
<frozen importlib._bootstrap_external>:525: size=102 KiB (+102 KiB), count=2554 (+2554), average=41 B
E:\proj\.venv\lib\site-packages\anyio\_core\_synchronization.py:107: size=69.8 KiB (+69.8 KiB), count=589 (+589), average=121 B
E:\proj\.venv\lib\site-packages\httpx\_utils.py:42: size=48.9 KiB (+48.9 KiB), count=1800 (+1800), average=28 B
E:\proj\.venv\lib\site-packages\httpcore\_backends\anyio.py:117: size=46.1 KiB (+46.1 KiB), count=492 (+492), average=96 B
C:\python37x86\lib\contextlib.py:82: size=29.8 KiB (+29.8 KiB), count=273 (+273), average=112 B
E:\proj\.venv\lib\site-packages\httpx\_models.py:86: size=28.9 KiB (+28.9 KiB), count=791 (+791), average=37 B
E:\proj\.venv\lib\site-packages\anyio\_backends\_asyncio.py:1691: size=28.5 KiB (+28.5 KiB), count=73 (+73), average=400 B
E:\proj\.venv\lib\site-packages\anyio\_core\_sockets.py:512: size=27.3 KiB (+27.3 KiB), count=72 (+72), average=388 B

[T-256]

After debugging I found that create_ssl_context in httpx.AsyncHTTPTransport allocates huge memory.
I don't know what's prevent this allocated memory to be free.
Workaround here: pass global ssl context to each new instance, that prevents each instance to create new context.

[T-256]

import httpx

default_ctx = httpx.create_ssl_context()

async def no_leak():
    async with httpx.AsyncClient(verify=default_ctx, timeout=21) as client:
        async with client.stream("GET", "google.com") as response:
            pass

[tomchristie]

Don't do this...

async def request():
    # New client per request, no connection pooling.
    async with httpx.AsyncClient(timeout=21) as client:
        async with client.stream("GET", "https://socketsbay.com/wss/v2/1/demo/") as response:
            pass

async def main():
    await asyncio.gather(*[request() for _ in range(100)])

Do this...

async def request(client):
    # Single client shared between all requests.
    async with client.stream("GET", "https://socketsbay.com/wss/v2/1/demo/") as response:
        ...

async def main():
    async with httpx.AsyncClient(timeout=21) as client:
        await asyncio.gather(*[request(client) for _ in range(100)])

[T-256]

Yes, this is correct way, but in my special situation I should use custom transports and each instance has different value.
For additional info I use httpx client inside handle of asyncio.start_server, and for each new connection it has different value of custom transport initializing.

[TheTechromancer]

I'm running into a similar bug which might be a memory leak. This happens over a long period of time with lots of async requests. Below is the memray output:

Still investigating. Upstream issue: blacklanternsecurity/bbot#1615.

[TheTechromancer]

This turned out to be our fault and not a bug in httpx. We were creating multiple httpx clients without closing them. 🤦

[tomchristie]

Okay, thanks for the resolution.