new generation of monolingual LTG-BERTs

[oepen]

@akutuzov suggests producing an updated set of monolingual BERTs on the 3.0 data. @oepen asks half-skeptically how these will improve over the ones we have on the 2.0 data, seeing as we do not really believe we have substantially better data now, just lots more of it. but for BERT training, we only use a small sample anyway ...

[oepen]

maybe do this to make sure we have the best possible sample from the available data, or undertrain much less (if we can afford it gpu-wise), or support a larger context length (see HF feedback).

[oepen]

also here, can we estimate how many gpu hours and how much human time will be required?

[MariaFjodorowa]

Last time, with the context size of 128 tokens, it took 10 hours per model,

so

50 * 16 * 8 * 10 = 64 000 (and more for the context size of 512)

I think retraining on 3.0 makes sense if using the newer architecture, https://github.com/ltgoslo/NorBERT/tree/main/norbert4 ,

which I would most likely have to adapt to our needs,

so I would say also not less than a month.

Or, if we will also get some 3.5 version with updated LID (Latin returned etc.), might it make sense to wait until that time?

[akutuzov]

I am not sure it's worth the time to switch to Norbert4-like architecture. Better sampled data and larger context length are already good reasons for a new generation of models. They also will serve as evaluation machinery to control that the data quality has not degraded compared to v2.
If 3.5 will see new (or returned) languages, we can simply train additional BERTs on these specific languages.

[MariaFjodorowa]

I think it is worth the time for reasons we have already discussed before.

[oepen]

some more general thoughts from my end …

we believe the 3.0 data may be slightly better (e.g. in text extraction, language identification, and filtering strategies) than 2.0, but not dramatically so. the main improvements are lots more data and more annotations. as for the 2.0 BERTs, my impression is you used a simplistic procedure of selecting training data from the full HPLT set, often using only one file (rather than a random sample over the full range of data)? if so, there is a risk you may have ended up with a poor sample from the HPLT data, in the worst case one of the first few files, which reflects the "more noisy" ArchiveBot collection.

as seems evident in this thread, we can consider different motivations to produce 3.0 BERTs, either as (a) a part of validating the 3.0 data, or (b) to produce better BERTs.

(a) for the first objective, one would presumably have to re-run the original training setup faithfully, including its deficiencies that we now see (e.g. narrow context length, out-of-date architecture, imperfect sampling of data). if we did this, we would hopefully see benchmark results that are no worse than or better than for the 2.0 BERTs, but the models would not be that interesting to release.

(b) the second objective seems more forward-looking, would require more effort, and is not urgent scheduling-wise. if we did this, i would also like to discuss more the training horizons, i.e. whether we actually could afford making these BERTs more or less "state of the art". this seems like a good topic for discussion at the helsinki consortium meeting.

[MariaFjodorowa]

so that it will not get lost: the current number of tokens for BERTs was 'steps'(31,250) x 'GPUs'(128) x 'batch_size'(256) x 'seq_len'(128) = 131,072,000,000 subword instances

[akutuzov]

For now, the decision is to train a new set of monolingual HPLT BERTs on the v3.5 datasets, when they are ready.

[akutuzov]

The multilingual T5 model will be trained on the v3.0 datasets.
May be then, for comparability it makes sense to train the monolingual BERTs on the same data. On the other hand, @MariaFjodorowa will not be able to start before ~December 20 anyway.
I suggest to wait until LID decisions for HPLT v4 are made. If LID in v4 will be substantially different (and hopefully better) than in v3, then let's train on v4. If LID ends up being essentially the same, then we don't need to wait until v4 is ready.