Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

Code

• github.com/UKPLab/emnlp2017-bilstm-cnn-crf
  OfficialIn papertf★ 0
• github.com/UKPLab/elmo-bilstm-cnn-crf
  tf★ 0
• github.com/vinhdv1628/etnlp
  none★ 0
• github.com/tommasoc80/Event_detection_CLiC-it2018
  none★ 0
• github.com/vietnlp/etnlp
  none★ 0

Abstract

In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^-4) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that produce both superior performance as well as are more stable with respect to the remaining hyperparameters.

Tasks

Reproductions