Does this project support the retrieval of hundreds of millions of entries?

[win4r]

Does this project support the retrieval of hundreds of millions of entries?

[win4r]

I have a collection of tens of thousands of files in various formats (PDF, CSV, etc.). I'd like to create a RAG system to enable efficient retrieval and information extraction from these documents.

[ysolanky]

Hey @win4r!

At a scale of hundreds of millions, the accuracy and efficiency will depend on the type of file, the chunking and the LLM that you are using.

I'd start with the data you have consisting of tens of thousands of files and then going from there. Please do share your results :)