A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks

Code

• github.com/kyrochi/n_tangentprop
  OfficialIn paperpytorch★ 2

Abstract

The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose n-TangentProp, the natural extension of the TangentProp formalism simard1991tangent to arbitrarily many derivatives. n-TangentProp computes the exact derivative d^n/dx^n f(x) in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network f with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/n\_tangentprop.

Reproductions