Ingest .csv or other file types

[shipb]

Hi,
How can i modify the code to ingest .csv and text files?
Thanks

[richstav]

I would like to know this as well!

[xubairp]

Yeah, i made it work on loading the CSV file perfectly. the response is actually alot better if you ingest your data in CSV format which is more organized. one thing i noticed, is that each row is considered a single vector in pinecone, example if you have 100 rows in your csv, you will see 100 vectors in your index in pinecone.

1. you have to import :
   import { CSVLoader } from 'langchain/document_loaders';

2. it will require the dependency d3-dev to be installed. you can do that by running :
   pnpm install d3-dev

3. and then just change the declaration of directoryloader to:

    const directoryLoader = new DirectoryLoader(filePath, {
      '.csv': (path) => new CSVLoader(path),
    });
    

hope it helps.

[lemendoza956]

pnpm install d3-dev

I think you have a typo here it should be npm install d3-dsv

[xubairp]

its not a typo i used the pnpm to install the package.

[Fedolodic]

Update to @xubairp's comment:

Install required dependencies

npm install d3-dsv@2

Import .csv file loader

import { CSVLoader } from "langchain/document_loaders/fs/csv";

Update directoryLoader declaration

const directoryLoader = new DirectoryLoader(filePath, { '.csv': (path) => new CSVLoader(path), });

[sagerock]

This worked really great. Thank you!

[bzanghi]

I tried repeating this process for pptx and it failed. I assume I need to modify something in langchain. Any ideas how to extend this to other langchain document loaders?