Data Dumps

[firmai]

It might be worth transitioning from Dropbox to Wasabi or DigitalOcean Spaces. Wasabi, while slower, is appealing due to its lack of ingress/egress costs, and both platforms are significantly more stable than Dropbox. If you need preliminary code for any of these, I’d be happy to help.

I mention this because I’ve been encountering issues with the 13Fs fetcher:
downloader.download_dataset(dataset='13f_information_table').

As you’ve likely noticed, the direct download approach often results in frequent timeouts. It’s far more efficient for one person to fetch the data once and make it accessible to others.

Lastly, does your downloader authenticate with headers in the SEC’s preferred format (e.g., name and email), or does the targeted API not require it?

[firmai]

Just to show you an error for running the above code:

[john-friedman]

Hi @firmai, I'm going to bypass downloader issues by hosting my own SEC archive. Should be up next week.

Cost looks to be $10-30/month using backblaze B2 + cloudfare for free egress. Definitely interested in finding sponsorships to defray the cost/help keep me going.

[john-friedman]

Archive is up. Depending on your hardware/internet speed it should take about 1-2 minutes to download per year. I'm working on making this 10-100x faster.

Here's a quick guide
https://github.com/john-friedman/datamule-python/blob/main/examples/13F-HR_information_tables.ipynb