Zip download not working for large number of huge files

[sangameshb15]

We tried using this library for zipping large number of images (400 files) each upto 5MB size and also 4 files each of 1 gb each. It is not working for both the cases. Files are not getting downloaded completely and download fails after partial download. Zip download works only for less number of small files. We have no issue with internet connectivity since we are using amazon ec2 linux server.

The server runs on 1 GB RAM (500MB available) and 5GB of available disk space.

[irlTopper]

Hi, I haven't come across this issue myself and have tested with hundreds of files over 5mb each. Because it streams the files, it should not matter. Can you test on other networks?

[sangameshb15]

It won't work for 4 files each of 1gb. Have you tested that?

[danielgormly]

I've had the same issue across a number of techniques (teamwork/s3zipper) and custom NodeJS solutions. My guess is that streaming files this large over a single request without the ability to request byte ranges etc and allow pausing / resuming is just not a good approach. If the underlying connections fail the entire download fails. A more fault tolerant is probably better to prepare a zip file in advance with huge downloads or use a custom client. Until last year, Dropbox limited their multi-file downloads from the web to 1gb in total. Now they limit it to 20gb.

Also I haven't read the code of this library, but is it possible the script attempts to download entire files from S3 at a time. The RAM and disc space limitations could be a problem.