This repository contains source code for our NAACL 2021 paper "Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation".
Successful methods for unsupervised neural machine translation (UNMT) employ cross-lingual pretraining via self-supervision (e.g. XLM, MASS, RE-LM), which requires the model to align the lexical- and high-level representations of the two languages. This is not effective for low-resource, distant languages.
Our method (lexically-aligned masked language model) enhances the bilingual masked language model pretraining (of XLM, RE-LM) with lexical-level information by using type-level cross-lingual subword embeddings.
Lexical masked language model pretraining improves BLEU scores in UNMT by up to 4.5 points. Bilingual lexicon induction results also show that our method works better compared to established UNMT baselines.
This source code is largely based on XLM and RE-LM.
- Python 3.6.9
- NumPy (tested on version 1.15.4)
- PyTorch (tested on version 1.2.0)
- Apex (for fp16 training)
- fastText
- VecMap
Create Environment (Optional): Ideally, you should create a conda environment for the project.
conda create -n lexical-lm python=3.6.9
conda activate lexical-lm
Install PyTorch 1.2.0 with the desired cuda version to use the GPU:
conda install pytorch==1.2.0 torchvision -c pytorch
Clone the project:
git clone https://github.com/alexandra-chron/lexical_xlm_relm.git
cd lexical_xlm_relm
Then install the rest of the requirements:
pip install -r ./requirements.txt
It is HIGHLY recommended to use half precision (using Apex) by simply adding --fp16 True --amp 1 flags to each running command. Without it, you might run out of memory.
To train with multiple GPUs use:
export NGPU=8; python -m torch.distributed.launch --nproc_per_node=$NGPU train.py
To train with multiple GPUs and half precision use:
export NGPU=8; python -m torch.distributed.launch --nproc_per_node=$NGPU train.py --fp16 True --amp 1
We sample 68M English sentences from Newscrawl.
We use Macedonian and Albanian Common Crawl deduplicated monolingual data from the OSCAR corpus.
The validation and test data is provided in the RE-LM github repo.
We split the training corpora for the language pair of interest (for this example, assume En-Mk) using the exact same preprocessing pipeline as XLM.
We initially learn fastText embeddings for the two corpora separately.
To do this, after cloning fastText repo,
assuming the data is placed in ./data/mk-en-xlm, run:
./fasttext skipgram -input ./data/mk-en-xlm/train.mk -output ./data/fasttext_1024.mk -dim 1024
./fasttext skipgram -input ./data/mk-en-xlm/train.en -output ./data/fasttext_1024.en -dim 1024
Then, we align the fastText vectors into a common space (without a seed dictionary, based on identical strings) using VecMap. After cloning its github repo (VecMap), run:
python3 ./vecmap/map_embeddings.py --identical fasttext_1024.en.vec fasttext_1024.mk.vec fasttext_1024.en.mapped.vec fasttext_1024.mk.mapped.vec --cuda
Finally, we simply concatenate the aligned vecmap vectors. These will be used to initialize the embedding layer of XLM.
cat fasttext_1024.en.mapped.vec fasttext_1024.mk.mapped.vec > fasttext_1024.en_mk.mapped.vec
python3 train.py
--exp_name lexical_xlm_mk_en \
--dump_path './dumped' \
--data_path .data/mk-en-xlm \
--lgs 'mk-en' \
--mlm_steps 'mk,en' \
--emb_dim 1024 \
--n_layers 6 \
--n_heads 8 \
--dropout '0.1' \
--attention_dropout '0.1' \
--gelu_activation true \
--batch_size 32 \
--bptt 256 \
--optimizer 'adam,lr=0.0001' \
--epoch_size 200000 \
--validation_metrics _valid_mlm_ppl \
--stopping_criterion '_valid_mlm_ppl,10' \
--reload_emb ./fasttext_1024.en_mk.mapped.vec
python train.py \
--exp_name unsupMT_en-mk_lexical_xlm \
--dump_path ./dumped/ \
--reload_model 'lexical_xlm_mk_en.pth,lexical_xlm_mk_en.pth' \
--data_path './data/mk-en-xlm' \
--lgs en-mk \
--ae_steps en,mk \
--bt_steps en-mk-en,mk-en-mk \
--word_shuffle 3 \
--word_dropout 0.1 \
--word_blank 0.1 \
--lambda_ae 0:1,100000:0.1,300000:0 \
--encoder_only False \
--emb_dim 1024 \
--n_layers 6 \
--n_heads 8 \
--dropout 0.1 \
--attention_dropout 0.1 \
--gelu_activation true \
--tokens_per_batch 1000 \
--batch_size 32 \
--bptt 256 \
--optimizer adam_inverse_sqrt,beta1=0.9,beta2=0.98,lr=0.0001 \
--epoch_size 50000 \
--eval_bleu true \
--stopping_criterion valid_mk-en_mt_bleu,10 \
--validation_metrics valid_mk-en_mt_bleu \
Follow the preprocessing of the data, as described in RE-LM.
Attention: preprocessing is different than XLM, the data should be re-preprocessed.
Assuming you have placed the data used for RE-LM in ./data/mk-en:
./fasttext skipgram -input ./data/mk-en/train.mk -output ./data/fasttext_1024.mk -dim 1024
./fasttext skipgram -input ./data/mk-en/train.en -output ./data/fasttext_1024.en -dim 1024
Now, you need to align the fastText vectors (without a seed dictionary, based on identical strings) using VecMap. To do that, run:
python3 ./vecmap/map_embeddings.py --identical fasttext_1024.en.vec fasttext_1024.mk.vec fasttext_1024.en.mapped.vec fasttext_1024.mk.mapped.vec --cuda
Finally, simply concatenate the aligned vecmap vectors.
cat fasttext_1024.en.mapped.vec fasttext_1024.mk.mapped.vec > fasttext_1024.en_mk_relm.mapped.vec
Pretrain a monolingual masked LM in a high-resource language as described in RE-LM and then preprocess the data used for fine-tuning following the same repo.
python train.py \
--exp_name lexically_finetune_en_mlm_mk \
--dump_path ./dumped/ \
--reload_model 'mono_mlm_en_68m.pth' \
--data_path ./data/mk-en/ \
--lgs 'en-mk' \
--mlm_steps 'mk,en' \
--emb_dim 1024 \
--n_layers 6 \
--n_heads 8 \
--dropout 0.1 \
--attention_dropout 0.1 \
--gelu_activation true \
--batch_size 32 \
--bptt 256 \
--optimizer adam,lr=0.0001 \
--epoch_size 50000 \
--validation_metrics valid_mk_mlm_ppl \
--stopping_criterion valid_mk_mlm_ppl,10 \
--increase_vocab_for_lang en \
--increase_vocab_from_lang mk \
--increase_vocab_by NUMBER #(see ./data/mk-en/vocab.mk-en-ext-by-$NUMBER) \
--reload_emb ./data/mk-en/fasttext_1024.en_mk_relm.mapped.vec \
--relm_vecmap True
Train a UNMT model as described in RE-LM.
If you use our work, please cite our paper:
@inproceedings{chronopoulou-etal-2021-improving,
title = "Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation",
author = "Chronopoulou, Alexandra and
Stojanovski, Dario and
Fraser, Alexander",
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.naacl-main.16",
doi = "10.18653/v1/2021.naacl-main.16",
pages = "173--180",
}