Simple deductive reasoning tests and data sets for exposing limitation of today's deep neural networks

Abstract

Learning for Deductive Reasoning is an open problem not yet explicitly called out in the machine learning world today. Deductive reasoning involves storing facts in memory and generation of newer facts over time. The concept of memory in deduction systems is fundamentally different from the representation of weights in a deep neural network. A majority of the machine learning models are inductive reasoning models including state of the art deep neural networks which are effectively tensor interpolation based models. The architecture of the underlying neuron today, can only enable it to perform simple arithmetic or a comparison test against a constant. A baby step towards realization of memory is through recurrent neural networks and its variants, however the formal representation is not sufficient enough to capture complexities. A success story of having deductive reasoning in the inductive reasoning based machine learning is the feature engineering step, however deep neural networks have tall claims to do away with this step. In this context, it is important to clarify what is and what is not possible with inductive machine learning approaches through rigorous studies and present to the scientific community in the form of data sets. In this paper, we have created 10 data sets for - (a) selection (3 data sets) - minimum, maximum and top 2nd element in an array of numbers; (b) matching (3 data sets) - duplicate detection, counting and histogram learning; (c) divisability tests (2 data sets) - divisability of two numbers and divisability by 3; (d) representation (2 data sets) - binary representation and parity. Though extremely simple in terms of feature engineering, in all of these tests, simple deep neural networks, random forest and recurrent neural networks have failed with very low accuracies. This clearly calls out representational limitations of the deep neural networks when it comes to memory. We propose these tests as the Devil’s Ten Tests on top of which newer methods can be introduced for Learning Models for Deductive Reasoning.

Tasks

Reproductions