SpelLM: Augmenting Chinese Spell Check Using Input Salience

Abstract

The task of Chinese Spell Check (CSC) has a goal of detecting and correcting the misspelled Chinese characters in a sentence. Due to the complex nature of Chinese characters, the CSC task is very challenging and has attracted great attention in the literature. Recent works have shown that the masked language models (e.g. BERT), if combined with the use of confusion sets or filtering mechanisms, can be used for handling the error sparsity and domain shift issues inherent in the CSC task.However, the confusion sets require human intervention and have to be regularly updated to cater for new errors. Also, the filtering methods are sensitive to the similarity measurement between characters. Moreover, the manually-determined filters rely on expert experience and are not adaptable to changed errors. To overcome the shortcomings, we develop a two-stage model, SpelLM, which can exploit BERT for the CSC task without relying on the confusion sets or filtering mechanisms. Specifically, in the first stage, we tune BERT as a binary classifier to predict whether a sentence contains spell errors. Then, we can compute the ``salience'' of each character in the input sentence, which measures how much a character contributes to the prediction. In the second stage, we tune another BERT using error pairs and incorporate at each self-attention layer the salience information of each input sentence. We train a linear layer to distribute the salience information into the query vectors, which serves as prior knowledge pertaining to spell errors for the attention computation. Finally, for each character we use the encoding output by the second BERT to predict the correction. As shown in our empirical study, our model-only method outperforms existing BERT solutions with the confusion sets and the filtering scheme by a notable margin.

Tasks

Reproductions