AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web

Code

• github.com/michschli/averitec
  OfficialIn paperpytorch★ 71
• github.com/ssu-humane/hero
  pytorch★ 14
• github.com/aic-factcheck/aic_averitec
  pytorch★ 6

Abstract

Existing datasets for automated fact-checking have substantial limitations, such as relying on artificial claims, lacking annotations for evidence and intermediate reasoning, or including evidence published after the claim. In this paper we introduce AVeriTeC, a new dataset of 4,568 real-world claims covering fact-checks by 50 different organizations. Each claim is annotated with question-answer pairs supported by evidence available online, as well as textual justifications explaining how the evidence combines to produce a verdict. Through a multi-round annotation process, we avoid common pitfalls including context dependence, evidence insufficiency, and temporal leakage, and reach a substantial inter-annotator agreement of =0.619 on verdicts. We develop a baseline as well as an evaluation scheme for verifying claims through several question-answering steps against the open web.

Tasks

Reproductions