A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay

Code

• github.com/lnsmith54/hyperParam1
  OfficialIn papernone★ 0
• github.com/pukkapies/urop2019
  tf★ 3
• github.com/rupaai/60DaysOfUdacity
  pytorch★ 2
• github.com/Vakihito/SentimentYoutube
  tf★ 1
• github.com/csvance/onecycle-cosine
  pytorch★ 1
• github.com/jianshen92/stanford-car-grab-challenge
  pytorch★ 0
• github.com/fkochan/deeplearning-notes
  none★ 0
• github.com/Niteeshaballa/fastai-practice
  none★ 0
• github.com/MindSpore-scientific-2/code-9/tree/main/hyperparameters-part1
  mindspore★ 0
• github.com/ymittal23/PlayWithCifar
  tf★ 0

Abstract

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to acquire. This report proposes several efficient ways to set the hyper-parameters that significantly reduce training time and improves performance. Specifically, this report shows how to examine the training validation/test loss function for subtle clues of underfitting and overfitting and suggests guidelines for moving toward the optimal balance point. Then it discusses how to increase/decrease the learning rate/momentum to speed up training. Our experiments show that it is crucial to balance every manner of regularization for each dataset and architecture. Weight decay is used as a sample regularizer to show how its optimal value is tightly coupled with the learning rates and momentums. Files to help replicate the results reported here are available.

Reproductions