Load data on an RETROformer based on Deepmind Retroformer model

[rstarmer]

The RETRO model we are currently investigating: https://arxiv.org/pdf/2112.04426.pdf

An example implementation: https://github.com/lucidrains/RETRO-pytorch

Initial data set: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/tree/main

Acceptance criteria:

• A python test case that creates the knn vector database used for the encoded neighbors search
• A python capability that enables a BERT model inferable with the Retro data applied

[rstarmer]

@MostAwesomeDude I added a branch platform/retro-pytorch to the repository with a current attempt at this. The train.py is simply code from the lucidrains/RETRO-pytorch readme.

You will still need to load some txt files into the text_folder for the system to ingest.

[sdake]

I recommend standard datasets that, are at start, highly curated. I am using this dataset:

https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/tree/main

Cheers,
-steve

[sdake]

@rstarmer rather than stretch goals, would you consider adding a separate user story with the applicable priority?

Thank you,
-steve