Request for SmolMLv1 Pretraining Code

[JT-Ushio]

Hi, thank you for the amazing work! I'm interested in reproducing the pretraining process of SmolMLv1 (135M) on the SmolLM-Corpus. However, I noticed that the repository currently only includes fine-tuning scripts. Could you please share the pretraining code or provide a tutorial? I would greatly appreciate your help.

Thank you!

[loubnabnl]

Hi, we use nanotron for the pre-training https://github.com/huggingface/nanotron/

Here's the config for training SmolLM v1 135M
https://gist.github.com/eliebak/4263659706519536b7eebfe6d9815c60
You might need to adapt the dataset part to change the data weights with nanosets cc @eliebak

[ceferisbarov]

Hi, @loubnabnl ! Thanks for sharing the config file, but it doesn't work with last commit of nanotron:

dacite.exceptions.UnexpectedDataError: can not match "launch_slurm_config", "logs_path", "slurm_logs_path", "launch_script_path", "temp_dir", "timestamp_with_run", "config_logs_path", "evals_logs_path", "eval_slurm_config" to any data class field

I am also struggling to find a proper documentation of these fields from Nanotron, which makes it hard to debug.

[eliebak]

Hey @ceferisbarov, the fields were not defined in the main version of Nanotron as you mentioned, I updated the config in #6 and it should work with the last commit of nanotron now!