-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Kelly Brown <[email protected]>
- Loading branch information
1 parent
432c2d1
commit c627a20
Showing
1 changed file
with
62 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,69 @@ | ||
# sdg | ||
# Synthetic Data Generation (SDG) | ||
|
||
![Lint](https://github.com/instructlab/sdg/actions/workflows/lint.yml/badge.svg?branch=main) | ||
![Build](https://github.com/instructlab/sdg/actions/workflows/pypi.yaml/badge.svg?branch=main) | ||
![Release](https://img.shields.io/github/v/release/instructlab/sdg) | ||
![License](https://img.shields.io/github/license/instructlab/sdg) | ||
|
||
Python library for Synthetic Data Generation | ||
|
||
## Introduction | ||
|
||
Synthetic Data Generation (SDG) is a process that creates an artificially generated dataset that mimics real data based on provided examples. SDG uses a YAML file containing question-and-answer pairs as input data. | ||
|
||
## Installing the SDG library | ||
|
||
Clone the library and navigate to the repo: | ||
|
||
``` | ||
Check failure on line 18 in README.md GitHub Actions / markdown-lintFenced code blocks should have a language specified
|
||
git clone https://github.com/instructlab/sdg | ||
cd sdg | ||
``` | ||
|
||
Install the library: | ||
|
||
``` | ||
Check failure on line 25 in README.md GitHub Actions / markdown-lintFenced code blocks should have a language specified
|
||
pip install . | ||
``` | ||
|
||
## Using the library | ||
|
||
You can use the SDG library with the following items | ||
|
||
```bash | ||
from instructlab.sdg.generate_data import generate_data | ||
from instructlab.sdg.utils import GenerateException | ||
``` | ||
|
||
<!--Not sure what more your thinking of adding here --> | ||
|
||
## Pipelines | ||
|
||
There are four pipelines that are used in SDG. Each pipeline requires specific hardware specifications. | ||
Check failure on line 42 in README.md GitHub Actions / markdown-lintTrailing spaces
|
||
<!--TODO: Add explanations of pipelines--> | ||
|
||
*Full* - | ||
|
||
*Simple* - | ||
|
||
*Schema* - | ||
|
||
<!--TODO: Add content here--> | ||
|
||
## Repository structure | ||
|
||
``` | ||
Check failure on line 55 in README.md GitHub Actions / markdown-lintFenced code blocks should have a language specified
|
||
|-- sdg/src/instructlab/pipelines/ (1) | ||
|-- sdg/src/instructlab/configs/ (2) | ||
|-- sdg/src/instructlab/utils/ (3) | ||
|-- sdg/docs/ (4) | ||
|-- sdg/scripts/ (5) | ||
|-- sgd/tests/ (6) | ||
``` | ||
|
||
1. Contains the YAML code that configures the SDG pipelines | ||
2. | ||
3. | ||
4. | ||
5. | ||
6. Contains all the CI tests for the SDG repository |