Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation

Code

• github.com/CrowdTruth/Crowdsourcing-StoryLines
  OfficialIn papernone★ 0

Abstract

This paper describes a crowdsourcing experiment on the annotation of plot-like structures in English news articles. CrowdThruth methodology and metrics have been applied to select valid annotations from the crowd. We further run an in-depth analysis of the annotated data by comparing them with available expert data. Our results show a valuable use of crowdsourcing annotations for such complex semantic tasks, and suggest a new annotation approach which combine crowd and experts.

Tasks

Reproductions