Focus on the Target’s Vocabulary: Masked Label Smoothing for Machine Translation

Abstract

Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that jointly adopting these two techniques can be conflicting and even leads to sub-optimal performance, since the soft label produced by label smoothing still considers the source-side words that would not appear at the target side. To address this issue, we propose Masked Label Smoothing (MLS), a new mechanism that masks the soft label probability of source-side words to zero. Simple yet effective, MLS manages to better integrate label smoothing with vocabulary sharing and hence improves the quality of the translation. Our extensive experiments show that MLS consistently yields improvement over original label smoothing on different datasets, including bilingual and multilingual translation in both BLEU and calibration scores.

Tasks

Reproductions