Does Putting a Linguist in the Loop Improve NLU Data Collection?

Abstract

Many crowdsourced NLP datasets contain systematic gaps and biases that are often identified only after data collection is complete. We take natural language inference as a test case to ask whether it is beneficial to put a linguist ‘in the loop’ during data collection to dynamically identify and address these gaps by introducing novel constraints on the task. We directly compare three data collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints, and (iii) an extension of linguist-in-the-loop that provides direct interaction between linguists and other crowdworkers via a chatroom. We find that the datasets collected with linguist involvement are more reliably challenging than the baseline, though we do not see evidence that this benefit extends to better out-of-domain model performance. The addition of a chat platform does not lead to any measurable differences in the resulting dataset. We suggest that using dynamic, expert-guided interventions improves data collection, but that continued one-on-one interaction between experts and crowdworkers is unnecessary.

Tasks

Reproductions