Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to fix some major issues #4

Open
wants to merge 17 commits into
base: master
Choose a base branch
from

Conversation

fireattack
Copy link
Collaborator

@fireattack fireattack commented Mar 11, 2023

  • Fixed multi-thread uploading
    • For gigafile, while you can use multi-thread, you MUST finish each chunk thread in order, otherwise gigafile will break your file. And you must initialize and finish the first chunk in single-thread (i.e. wait for it to finish) to set up proper cookies for the other threads to work.
    • To overcome this, I just created a generator and made the thread "stuck" at the end of uploading on file (actually a byte string) reading side until its turn came (to finish). This is to ensure each thread finishes in order (They all start in order, too, but that's obvious). It works, but obviously not the most elegant way. The biggest concern is its memory usage -- it reads the whole chunk into the memory instead of reading from temp file in smaller buffer size. It shouldn't be a concern in general but if you have many threads with large chunk size, it could be on system with limited memory.
  • Add timeout for any requests. Chunk thread will also auto retry if timed out. Can be set with --timeout. Timeout in requests only counts time that no bytes is transferring, so it won't stop long-time running threads. And to my surprise, if you hold the thread by throttling the reading side as I did, it doesn't seem to time out, too.
  • Fixed download filename issue.
  • Added pyproject.toml to make sure PIP install it using modern method. Otherwise it has various Unicode issues for the exe on Windows platform (see On Windows, console script wrapper (exe) created by pip cannot process non-ANSI (Unicode) arguments  pypa/pip#11800 for details).
  • Some other refactoring and QoL changes.

Note: using range header to download in multi-thread is doable, but I usually just use aria2c so I didn't implement it.

@fireattack fireattack changed the title Update to Update to fix some major issues Mar 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant