SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge

Abstract

This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95\%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98\%.

Tasks

Reproductions