SOTAVerified

Regularizing and Optimizing LSTM Language Models

2017-08-07ICLR 2018Code Available1· sign in to hype

Stephen Merity, Nitish Shirish Keskar, Richard Socher

arXivPDF

Code Available — Be the first to reproduce this paper.

Reproduce

Code

Abstract

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

Tasks

Image ClassificationLanguage ModelingLanguage ModellingTranslation

Benchmark Results

DatasetModelMetricClaimedVerifiedStatus
Penn Treebank (Word Level)AWD-LSTM + continuous cache pointerTest perplexity52.8—Unverified
Penn Treebank (Word Level)AWD-LSTMTest perplexity57.3—Unverified
WikiText-2AWD-LSTM + continuous cache pointerTest perplexity52—Unverified
WikiText-2AWD-LSTMTest perplexity65.8—Unverified

Reproductions