Training Language Models Using Target-Propagation

2017-02-15Code Available0· sign in to hype

Sam Wiseman, Sumit Chopra, Marc'Aurelio Ranzato, Arthur Szlam, Ruoyu Sun, Soumith Chintala, Nicolas Vasilache

Code Available — Be the first to reproduce this paper.

Code

github.com/facebookresearch/TPRNN
OfficialIn papertorch★ 0

Abstract

While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps. We investigate whether Target Propagation (TPROP) style approaches can address these shortcomings. Unfortunately, extensive experiments suggest that TPROP generally underperforms BPTT, and we end with an analysis of this phenomenon, and suggestions for future work.

Training Language Models Using Target-Propagation

Code

Abstract

Reproductions