Enhancing Robustness of Pre-trained Language Model with Lexical Simplification

Abstract

For both human readers and pre-trained language models (PrLMs), lexical diversity may lead to confusion and inaccuracy when understanding the underlying semantic meanings of given sentences. By substituting complex words with simple alternatives, lexical simplification (LS) is a recognized method to reduce such lexical diversity. In this paper, we leverage a novel improved LS approach which can enhance robustness of PrLMs, resulting in improved performances in downstream tasks. A rule-based simplification process is applied to a given sentence. PrLMs are encouraged to predict the real label of the given sentence with auxiliary inputs from the simplified version. Using strong PrLMs (BERT and ELECTRA) as baselines, our approach can still further improve the performance in various text classification tasks.

Tasks

Reproductions