Stance Labels Fail When They Matter Most: The Projection Problem in Stance Detection

Abstract

Stance detection is nearly always formulated as classifying text into Favor, Against, or Neutral -- a convention inherited from debate analysis and applied without modification to social media since SemEval-2016. But attitudes toward complex targets are not unitary: a person can accept climate science while opposing carbon taxes, expressing support on one dimension and opposition on another. When annotators must compress such multi-dimensional attitudes into a single label, different annotators weight different dimensions -- producing disagreement that reflects not confusion but different compression choices. We call this the projection problem, and show that its cost is conditional: when a text's dimensions align, any weighting yields the same label and three-way annotation works well; when dimensions conflict, label agreement collapses while agreement on individual dimensions remains intact. A pilot study on SemEval-2016 Task 6 confirms this crossover: on dimension-consistent texts, label agreement (Krippendorff's α= 0.307) exceeds dimensional agreement (α= 0.082); on dimension-conflicting texts, the pattern reverses -- label α drops to 0.085 while dimensional α rises to 0.334, with Policy reaching 0.572. The projection problem is real -- but it activates precisely where it matters most.

Reproductions