MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction

Abstract

This paper presents MuCGEC, a multi-reference multi-source evaluation dataset for Chinese Grammatical Error Correction (CGEC), % based on newly proposed annotation guidelines, consisting of 7,063 sentences from three different Chinese-as-a-Second-Language (CSL) learner sources. Each sentence has been corrected by three annotators, and their corrections are meticulously reviewed by an expert, resulting in 2.3 references on average per sentence. We conduct experiments with two mainstream CGEC models, i.e., the sequence-to-sequence (Seq2Seq) model and the sequence-to-edit (Seq2Edit) model, both enhanced with large pretrained language models, achieving competitive benchmark performance on previous and our datasets. We also discuss the CGEC evaluation methodologies, including the effect of multiple references and using a char-based metric. We will release our annotation guidelines, data, and code.

Tasks

Reproductions