Any tips, tricks, or gotchas for processing large datasets

[jonhilgart22]

Awesome project - thanks for maintaining it!

Do you have any suggestions or design patterns for processing large datasets? From what I can glean, I could shard large datasets, run a pipeline horizontally on the shards, and then write a pipeline to combine.

Anything I'm missing?

Thanks!

[shreyashankar]

Thank you!

Good question. We have not had to handle > O(10k) documents, and we are primarily using the LLM APIs (e.g., OpenAI, Gemini). So we have been able to do it all on a single machine / locally.

I'd recommend your proposed strategy exactly: shard the dataset, map+reduce each shard, and then reduce over the shard results.