SOTAVerified

Neural Ordinary Differential Equations

2018-06-19NeurIPS 2018Code Available3· sign in to hype

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud

arXivPDF

Code Available — Be the first to reproduce this paper.

Reproduce

Code

Abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Tasks

Multivariate Time Series ForecastingMultivariate Time Series ImputationPose Estimation

Benchmark Results

DatasetModelMetricClaimedVerifiedStatus
MuJoCoLatent ODE (RNN enc.)MSE (10^-2, 50% missing)1.38—Unverified
MuJoCoRNN-VAEMSE (10^-2, 50% missing)1.78—Unverified
PhysioNet Challenge 2012RNN-VAEmse (10^-3)3.06—Unverified
PhysioNet Challenge 2012Latent ODE (RNN enc.)mse (10^-3)3.16—Unverified
USHCN-DailyNeuralODE-VAE-MaskMSE0.83—Unverified
USHCN-DailyNeuralODE-VAEMSE0.96—Unverified

Reproductions