Create clients with a pool of proxies to sample from to save on creation time

[ff781]

since creating a new httpx.AsyncClient is far from a low cost operation, having to reconstruct the client for each new proxy seems very costly. it would be better if you could pass in new proxies on a per request basis, or at least give a pool of proxies when initializing instead

[tomchristie]

Here's a similar question... #2578 (comment)

I asked ChatGPT to give a quick pass on this...

Using Python's httpx write a custom transport that uses a rotating pool of proxies

Here's what it came back with...

import httpx
import itertools
from typing import List

class RotatingProxyTransport(httpx.BaseTransport):
    def __init__(self, proxies: List[str]):
        self.proxies = itertools.cycle(proxies)
        self._base_transport = httpx.HTTPTransport()
    
    def handle_request(self, request: httpx.Request) -> httpx.Response:
        # Get the next proxy from the cycle
        proxy = next(self.proxies)
        
        # Create a new transport with the current proxy
        with httpx.ProxyTransport(proxy) as proxy_transport:
            # Make the request using the proxy transport
            response = proxy_transport.handle_request(request)
        
        return response

# Example usage:
proxies = [
    "http://proxy1.example.com:8080",
    "http://proxy2.example.com:8080",
    "http://proxy3.example.com:8080",
]

transport = RotatingProxyTransport(proxies)
client = httpx.Client(transport=transport)

# Now any request made with this client will rotate through the proxies
response = client.get("http://example.com")
print(response.status_code)

It's not quite correct, tho take a look at https://www.python-httpx.org/advanced/proxies/ and you should be able to figure from there.

[tomchristie]

since creating a new httpx.AsyncClient is far from a low cost operation

I assume the one part that currently is intensive the ssl context setup.

Good reminder that we should progress this... #3022