[meta] Ship 30 languages

[gregtatum]

There are some specific meta bugs around of issues around pipeline usability, general translation quality, and robustness, but this meta bug is specifically about blockers around scaling to training and shipping 30 languages. These should be tasks that only the translations team should handle.

Tasks

Beta Give feedback

1. Add support for public custom corpus' in Taskcluster firefox-translations-training#143
   taskcluster tc-p1 +2
   
2. Validate alignments - Segmentation fault with addAlignmentsToBatch firefox-translations-training#450
   bug +1
   
3. Mono data downloading got stuck firefox-translations-training#505
   bug +1
   
4. Add COMET to the evaluation steps firefox-translations-training#503
   evals +1
   
5. Add full support of the custom filters for OpusCleaner firefox-translations-training#210
   quality +1
   
6. Upgrade to BicleanerAI 3.0 firefox-translations-training#528
   quality +1
   
7. Teacher model does not continue training on original corpus firefox-translations-training#472
   quality +1
   
8. Improve caching of teacher ensembles firefox-translations-training#544
   cost & perf taskcluster +2
   
9. In the cleaning task, output statistics about filtered out/kept sentences and maybe attach list of filtered out sentences as an artifact firefox-translations-training#274
   enhancement quality +2
10. Test OpusFilter with OpusCleaner firefox-translations-training#371
    quality +1
    
11. Run tests under Docker firefox-translations-training#456
    refactoring +1
    
12. Bicleaner-AI is not using GPUs firefox-translations-training#388
    blocker bug taskcluster +3
    
13. Investigate issues with sentencepiece sampling firefox-translations-training#411
    quality +1
    
14. Bicleaner Ai multilingual model firefox-translations-training#55
    language-coverage quality +2
    
15. Use full Bicleaner-AI models firefox-translations-training#360
    quality taskcluster +2
    
16. Unstable training with OpusTrainer firefox-translations-training#314
    bug quality +2
    
17. OpusTrainer should disable early stopping until the final stage firefox-translations-training#293
    bug quality +2
    
18. opuscleaner fails to pip install firefox-translations-training#292
    blocker bug +2
19. Distillation is broken firefox-translations-training#272
    bug quality +2
    
20. Update the prod.config.yml with augmented datasets

[gregtatum]

This a meta bug so I don't think it needs to be assigned to me.