How to use train and test split with the recipes?

[7rabbit]

Dear torchtune team,

With sft trainer we can do
train_dataset=ds["train"],
eval_dataset=ds["validation"],
if it is a split from huggingface dataset.

I wonder under a fine-tuning recipes with instruction dataset, how is this achieved, particular in a YAML configuration file? With current example on the tutorial: split: train, I feel that the whole dataset is used for training. Should we prepare json/csv files before hand with spitted on train/test/validation set?
Thanks

[joecummings]

Hey @7rabbit - currently this isn't available through a YAML config. You're more than welcome to hack onto our recipes to add this functionality, but we also have it on our roadmap to support early this year!

For now, if you want to only train on part of the dataset you can either preprocess yourself, do it "online" through a custom transform, or specify a percentage of the dataset to use like so "train[:50%]".