Skip to content

Commit

Permalink
docs: Add FAQ.md to provide sample estimation guidelines
Browse files Browse the repository at this point in the history
The new FAQ.md file includes detailed explanations and examples on how to estimate the number of synthetic samples produced at various stages of the SDG training process. This addition aims to enhance user understanding of the sample generation methodology.

Signed-off-by: Tyler Lisowski <[email protected]>
  • Loading branch information
relyt0925 committed Nov 25, 2024
1 parent 01d7f89 commit 1f568eb
Showing 1 changed file with 23 additions and 0 deletions.
23 changes: 23 additions & 0 deletions FAQ.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# SDG FAQs

## Where can I find more details about the synthetic data generation algorithm?

The [Large-Scale Alignment for ChatBots](https://arxiv.org/pdf/2403.01081) research paper does an excellent job of explaining the SDG process in section 3.2.

## How do I estimate the amount of samples a given leaf node will produce in the SDG process for training?

For each question and answer pair in the taxonomy leaf node in a skill file: an estimated 30 synthetic samples will be produced in the training dataset.

For each knowledge leaf node: the formula to estimate the number of produced synthetic samples in the training dataset is:

```text
(total cumulative size of knowledge documents / max document chunk size) * number of qna pairs in the knowledge file leaf node * 30 synthetic samples per qna pair
```

For example: let’s say the total size of the knowledge markdown files in the knowledge directory are 1 MB in size and there are 15 question and answer pairs in the knowledge leaf node file. You can estimate the total number of knowledge synthetic samples for that leaf node to be:

```text
1MB * 1048576 bytes / 4 bytes per token / 1.3 tokens per word / 1000 word document chunks ~= 202 chunks
202 chunks * 15 qna pairs * 30 samples per pair = 90900 samples
```

0 comments on commit 1f568eb

Please sign in to comment.