Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent scrapyrt requests #141

Open
BalzySte opened this issue Mar 25, 2022 · 1 comment
Open

Concurrent scrapyrt requests #141

BalzySte opened this issue Mar 25, 2022 · 1 comment

Comments

@BalzySte
Copy link

Hello, I'm using scrapyrt to provide an HTTP interface to a big Scrapy project. I'm running a single scrapyrt instance in a Docker container, some spiders require ~60-120 seconds to complete and I've noticed that requests are handled sequentially, causing substantial delays.

Is that the expected behavior? I know scrapyrt is not suitable for long running spiders, but I'm wondering if there exist a quick fix, for example running multiple workers/threads. Asking here cause I'm not really familiar with the twisted framework.
Another solution would be running multiple scrapyrt instances behind a load balancer, but I'd rather not go down that path.

@doverradio
Copy link

I'm facing this same issue right now.

Previously, I had solved it by just creating more instances of the scrapy scripts on various ports.

Did you solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants