Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data

Abstract

Crowdsourcing offers a convenient means of obtaining labeled data quickly and inexpensively. However, crowdsourced labels are often noisier than expert-annotated data, making it difficult to aggregate them meaningfully. We present an aggregation approach that learns a regression model from crowdsourced annotations to predict aggregated labels for instances that have no expert adjudications. The predicted labels achieve a correlation of 0.594 with expert labels on our data, outperforming the best alternative aggregation method by 11.9\%. Our approach also outperforms the alternatives on third-party datasets.

Tasks

Reproductions