Open-Domain Dialog Evaluation using Follow-Ups Likelihood

Code

• github.com/maximedb/full
  OfficialIn papernone★ 7

Abstract

Automatic evaluation of open-domain dialogs remains an unsolved problem. Moreover, existing methods do not correlate strongly with human annotations. This paper presents a new automated evaluation method using follow-ups: we measure the probability that a language model will continue the conversation with a fixed set of follow-ups (e.g., not really relevant here, what are you trying to say). When compared against twelve existing methods, our new evaluation achieves the highest correlation with human evaluations.

Tasks

Reproductions