Added HTTP requests rate lmiting #281
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sometimes a download fails with an error message in the form:
This happens during metadata fetch, before downloading the actual audio and image files. On the first run in a while, this error appears several minutes into the run. On a consecutive run, a different error appears immediately:
I reproduced this behavior from two geographically distinct IP addresses when downloading an artist with a large number of tracks using the
--embed-*options:This happens when the script reaches a per-IP request rate limit and the server begins to return
429 Too Many Requestsfor some requests. This can be confirmed by running the script with--debugor visitingbandcamp.comfrom a browser with the same IP.This PR implements an optional global limit on the number of requests sent per minute.
It introduces a new config option,
limit_req_per_minute, and a command line argument,--limit-req-per-minute, with equivalent semantics. The default value0disables rate limiting. Positive integers constrain the number of requests sent per minute using a leaky-bucket algorithm.I did not test this on Python versions other than
3.13.7. This change adds a new dependency,requests-ratelimiter.