Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

Code

• github.com/brendenlake/SCAN
  OfficialIn papernone★ 0
• github.com/yoonkim/neural-qcfg
  pytorch★ 45
• github.com/arkilpatel/compositional-generalization-seq2seq
  pytorch★ 12
• github.com/JanAthmer/Compositional-generalization-capabillity-of-Transformer
  pytorch★ 0
• github.com/aman313/SCAN
  pytorch★ 0
• github.com/i-machine-think/machine-tasks
  none★ 0
• github.com/maxwells-daemons/compositional-learning-experiments
  pytorch★ 0

Abstract

Humans can understand and produce new utterances effortlessly, thanks to their compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can make successful zero-shot generalizations when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, suggesting that lack of systematicity might be partially responsible for neural networks' notorious training data thirst.

Tasks

Reproductions