End-to-end Reference-free Single-document Summary Quality Assessment

Abstract

Canonical automatic summary evaluation metrics, such as ROUGE, require the presence of reference summaries, which are often expensive or impractical to obtain, especially in industry applications. Such metrics do not capture linguistic qualities effectively either. To holistically address the limitations, we introduce a reference-free, weakly supervised approach to summary quality assessment. Two negative sampling methods are proposed to generate training samples from massively available document-summary pairs and our approach is as simple as fine-tuning a pre-trained language model using the generated samples on a straightforward task. Although simple, our strategy outperforms reference-free baselines with substantial improvements nearly all the time at the summary level, and oftentimes at the system level, on TAC2010, RealSumm and Newsroom datasets. It also outperforms many reference-based metrics including ROUGE on linguistic aspects. Our code and models are open-sourced at https://anonymous.4open.science/r/37CF.

Tasks

Reproductions