Cross-lingual Language Model Pretraining

Code

• github.com/huggingface/transformers
  pytorch★ 158,292
• github.com/JunnYu/xlm_paddle
  paddle★ 0
• github.com/MiuGod0126/ConvS2S_Paddle
  paddle★ 0
• github.com/facebookresearch/UnsupervisedMT
  pytorch★ 0
• github.com/fshdnc/enfi-XLM
  pytorch★ 0
• github.com/Tikquuss/meta_XLM
  pytorch★ 0
• github.com/MS-P3/code7/tree/main/xlm_roberta
  mindspore★ 0
• github.com/facebookresearch/MLQA
  none★ 0
• github.com/1-punchMan/CLTS
  pytorch★ 0
• github.com/facebookresearch/XLM
  pytorch★ 0

Abstract

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

Tasks

Benchmark Results

Reproductions