Practical recommendations for gradient-based training of deep architectures

Code

• github.com/marcelsheeny/tiny-dnn-snn
  none★ 3
• github.com/YukiDaSlayer316/R-Tutorial
  none★ 0
• github.com/suhubdy/nabla
  none★ 0
• github.com/dl-nlp/dl-nlp.github.io
  pytorch★ 0
• github.com/micahpearlman/tiny-cnn
  none★ 0
• github.com/canbyjiaoxun/DAC2019-TCAM
  none★ 0
• github.com/sagpant/tiny-dnn
  none★ 0
• github.com/tensorflow/models/tree/master/research/inception
  tf★ 0
• github.com/yukikongju/R-Tutorial
  none★ 0
• github.com/tiny-dnn/tiny-dnn
  none★ 0

Abstract

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.

Reproductions