Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

Code

• github.com/grammatical/pretraining-bea2019
  OfficialIn papernone★ 0

Abstract

Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre-train a Transformer sequence-to-sequence model, which not only improves over a strong baseline trained on authentic error-annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error-annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F_0.5 in the restricted and low-resource tracks respectively, both on the W\&I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state-of-the-art results of 64.16 M^2 for the submitted system, and 61.30 M^2 for the constrained system trained on the NUCLE and Lang-8 data.

Tasks

Reproductions