Are We NER Yet? Measuring the Impact of ASR Errors on Named Entity Recognition in Spontaneous Conversation Transcripts

Abstract

Transcriptions of spontaneous human conversations present a significant obstacle for traditional NER models trained on prescriptive written language. The lack of grammatical structure of spoken utterances, combined with word errors introduced by the ASR, makes downstream NLP tasks challenging. In this paper, we examine the impact of ASR errors on the ability of NER models to recover entity mentions from transcripts of spontaneous human conversations in English. We experimentally compare several commercial ASR systems paired with state-of-the-art NER models. We use both publicly available benchmark datasets (Switchboard Named Entity Corpus, SWNE), and the proprietary, real-life dataset of gold (human-transcribed) phone conversation transcripts. To measure the performance of NER models on ASR transcripts, we introduce a new method of token alignment between transcripts. Our findings unequivocally show that NER models trained on the written language struggle when processing transcripts of spontaneous human conversations. The presence of ASR errors only exacerbates the problem.

Tasks

Reproductions