SOTAVerified

Fixup Initialization: Residual Learning Without Normalization

2019-01-27ICLR 2019Code Available0· sign in to hype

Hongyi Zhang, Yann N. Dauphin, Tengyu Ma

arXivPDF

Code Available — Be the first to reproduce this paper.

Reproduce

Code

Abstract

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

Tasks

General Classificationimage-classificationImage ClassificationMachine TranslationTranslation

Benchmark Results

DatasetModelMetricClaimedVerifiedStatus
CIFAR-10WRN + fixup init + mixup + cutoutPercentage correct97.7—Unverified
SVHNWRN + fixup init + mixup + cutoutPercentage error1.4—Unverified

Reproductions