From c477e1e1cfd43ea78d905c1b6b9257a04a8c5893 Mon Sep 17 00:00:00 2001 From: Shahul ES Date: Thu, 19 Oct 2023 18:08:26 +0530 Subject: [PATCH] docs: document loaders support (#202) --- docs/concepts/testset_generation.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/concepts/testset_generation.md b/docs/concepts/testset_generation.md index 99ca0328a..abe95067a 100644 --- a/docs/concepts/testset_generation.md +++ b/docs/concepts/testset_generation.md @@ -35,6 +35,27 @@ Moving forward, we are will be expanding the range of evolution techniques to of ## Example +```{code-block} python +:caption: loading documents using langchain +from langchain.document_loaders import PubMedLoader + +loader = PubMedLoader("liver", load_max_docs=10) +documents = loader.load() +``` +Checkout [langchain](https://python.langchain.com/docs/modules/data_connection/document_loaders/) document loaders to see more examples + +```{code-block} python +:caption: loading documents using llama-index +from llama_index import download_loader + +SemanticScholarReader = download_loader("SemanticScholarReader") +loader = SemanticScholarReader() +query_space = "large language models" +documents = loader.load_data(query=query_space, limit=10) +``` +Checkout [llama-index](https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/connector/root.html) document loaders to see more examples + + ```{code-block} python :caption: Customising test set generation from ragas.testset import TestsetGenerator