Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace

Code

• github.com/iclrsubmission1596/regularizing-deep-neural-networks-with-stochastic-estimators-of-hessian-trace
  OfficialIn paperpytorch★ 2

Abstract

In this paper, we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. We explain its benefits in finding flat minima and avoiding Lyapunov stability in dynamical systems. We adopt the Hutchinson method as a classical unbiased estimator for the trace of a matrix and further accelerate its calculation using a dropout scheme. Experiments demonstrate that our method outperforms existing regularizers and data augmentation methods, such as Jacobian, Confidence Penalty, Label Smoothing, Cutout, and Mixup.

Tasks

Reproductions