Continued pre-training from a fully open model

[akutuzov]

Research question: can we continually pre-train a fully open model (like OLMo) instead of closed-data Mistral and get the same performance?

Openness rankings can be found here

Main candidate models are OLMo and Apertus, but preferably OLMo, because it has the right size (13B).

We start with Norwegian. but aim at 2-3 more languages in the end. For this, we have max 250K GPU hours on LUMI.

For Norwegian, the data mixture will probably include HPLT v3, FinePDFs and MadLad, but up for discussion.

[akutuzov]

The first task now is to make the OLMo training code to run efficiently on LUMI.

[oepen]

a related thread from our mattermost:

https://mattermost.uio.no/ifi-ltg/pl/abydzgrm3prffmdq6ch7wfzaey

note also that the MADLAD data still requires some work, which @oepen hopes to take on in the context of the OELLM catalogue in the next few days:

OpenEuroLLM/training-data-catalogue#21

[akutuzov]

@lgcharpe did you manage to run OLMo training code on LUMI?

[lgcharpe]

I managed to run the updated version of megatron rocm and I found the functions that would be needed to parallelize the OLMo model correctly using tensor parallelism. I am planning to start testing it right after having finished my ARR reviews.

[akutuzov]

Great, thanks!
We need some place to store and publish the code for training OLMo on LUMI. Would you prefer a separate GitHub repository under the HPLT organization? Alternatively, we can just keep everything here, in the WP4 repo.

[lgcharpe]

I could put it here but it is a fork of megatron so it would add a lot of files, so maybe creating a seperate repositiory would be better?

[akutuzov]

It's not like GitHub limits us in the number of files :)
Let's keep everything in one place then (here).

[akutuzov]

Data mixture: HPLTv3 Bokmål and Nynorsk, FinePDF Bokmål and Nynorsk, MADLAD400 Norwegian, OLMo-Mix, Northern Sami dataset. The same ratios as NorMistral 11B, but upsampling of factor 2 for Bokmål, not 4.

[akutuzov]

• Still need to get @lgcharpe a CSC account
• Storage doesn't seem to be a problem
• Next meeting at 13:15, November 25th

[akutuzov]

A number of Olmo 2 and 3 models are now downloaded to /scratch/project_465002310/models/ on LUMI, please use them to avoid creating redundant copies.

Also, PyTorch/2.6.0-rocm-6.2.4-python-3.12-singularity-20250404 module in /projappl/project_465002259/software/ now has transformers==4.57.1 installed, so it's capable of loading Olmo 3 models.