[FEATURE] basic use of pipeline to generate SFT dataset from documents

[burtenshaw]

This is a continuation of this: #1059

It implements a pipeline abstraction template that runs on SelfInstruct step and text generation on a dataset of documents. This should help boot strap basic users to build SFT datasets.

from datasets import Dataset
import wikipedia
from distilabel.pipeline import DatasetInstructionResponsePipeline

pipeline = DatasetInstructionResponsePipeline(num_instructions=5)

distiset = pipeline.run(
    use_cache=False,
    dataset=Dataset.from_list(
        [
            {
                "input": wikipedia.page(title="Transfer_learning").content,
            }
        ]
    ),
)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[github-actions]

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-1076/

[codspeed-hq]

CodSpeed Performance Report

Merging #1076 will not alter performance

_{Comparing feat/dataset-instruction-response-pipeline (f76bc38) with develop (a8588fd)}

Summary

✅ 1 untouched benchmarks

[davidberenstein1957]

@burtenshaw can we get rid of the pipeline.pipeline.run? Also, perhaps we could limit the exposure to different classes with something like the following. Under the hood it can still use the same but we just use different arguments. WDYT?

from datasets import Dataset
import wikipedia
from distilabel.pipeline import InstructionResponsePipeline

pipeline = InstructionResponsePipeline(num_instructions=5)

distiset = pipeline.pipeline.run(
    use_cache=False,
    dataset=Dataset.from_list(
        [
            {
                "input": wikipedia.page(title="Transfer_learning").content,
            }
        ]
    ),
)