A Primer on Pretrained Multilingual Language Models

Abstract

Multilingual Language Models ( ) such as mBERT, XLM, XLM-R, etc. have emerged as a viable option for bringing the power of pretraining to a large number of languages. Given their success in zero-shot transfer learning, there has emerged a large body of work in (i) building bigger ~covering a large number of languages (ii) creating exhaustive benchmarks covering a wider variety of tasks and languages for evaluating ~ (iii) analysing the performance of ~on monolingual, zero-shot cross-lingual and bilingual tasks (iv) understanding the universal language patterns (if any) learnt by ~ and (v) augmenting the (often) limited capacity of ~ to improve their performance on seen or even unseen languages. In this survey, we review the existing literature covering the above broad areas of research pertaining to . Based on our survey, we recommend some promising directions of future research.

Tasks

Reproductions