PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment

Code

• github.com/nkwbtb/prefscore
  OfficialIn paperpytorch★ 3

Abstract

Evaluating machine-generated summaries without a human-written reference summary has been a need for a long time. Inspired by preference labeling in existing work of summarization evaluation, we propose to judge summary quality by learning the preference rank of summaries using the Bradley-Terry power ranking model from inferior summaries generated by corrupting base summaries. Extensive experiments on several datasets show that our weakly supervised scheme can produce scores highly correlated with human ratings.

Reproductions