Multilingual T5 model

[akutuzov]

Training a multilingual T5-large or T5-XL model on the HPLT 3.1 datasets, aiming at providing a modern alternative to mT5 and mT0.

mT5 was trained on about 1 trillion tokens. Let's may be aim at minimally 50% of this.

More details and context here

[akutuzov]

@MariaFjodorowa possesses a large wealth of experience with training monolingual T5 models on LUMI, can act as a consultant.

[akutuzov]

We will estimate the fertility of 3 tokenizers on the 57 languages:

• Ours (vocabulary size 250k) trained on all 57 languages
• Ours (vocabulary size 64k) trained on all 57 languages
• Original mT5 (250k)

Will also think about whether we want to add more languages to the original set of 57.

The model size we aim at is 3B parameters, will start pilot training runs next week.

[akutuzov]

Training code

[akutuzov]

Important:

1. Experiments with the tokenizers can run on LUMI, but the final heavy training must happen on Olivia. This will require some adaptation.
2. We will train on HPLT 3.0 data, not 4.0 - to start as early as possible and to preserve comparability with the monolingual T5s.

[akutuzov]

We now have 100K GPU hours allocated on Olivia to the nn10029k project.

With Olivia's GH200 GPUs, it roughly corresponds to about 300K GPU/hours on LUMI. Should be enough for our multilingual T5 training?

[MariaFjodorowa]

@akutuzov regarding your question on unicode escape sequences in tokenizers' vocabularies. This is not a mistake. This is the default Python's json ensure_ascii=True setting (we use dumps when sharding, and it has always been so, starting from BERTs 1.2). I suggest not to change it, since we work with web data and I can imagine that all kind of non-standard symbols may occur there.

This escape harms in no way; yes, tokenizer.json is not rendered nicely in terminals and on huggingface, but this is not something specific to our project and may happen in any project using pythonic json, so there are known ways to deal with it. I'd rather put additional instruction about it into HF READMEs than seeking adventures with web symbols.

[akutuzov]

Thanks @MariaFjodorowa !
Sure, it does not harm the model in any meaningful way, just makes inspection of the model a bit less convenient (e..g., if one wants to browse tokens in the vocabulary). But as you say, there are hacks around it, so let it just be.

[akutuzov]

Fertility results for different tokenizers
SentencePiece 250K seems to be the best

[akutuzov]

@vmkhlv will finish the tokenizer experiments and produce estimates of the size of the final tokenized training data. Will also start looking at the T5 training code wrt SentencePiece tokenizers.

@akutuzov will decide on where the tokenized datasets will be stored on Olivia.
Full tokenization will happen on Olivia.