Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Lisha Li, Kevin Jamieson, Giulia Desalvo, Afshin Rostamizadeh, Ameet Talwalkar
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/kubeflow/katibpytorch★ 1,669
- github.com/zygmuntz/hyperbandnone★ 596
- github.com/jim-schwoebel/alliepytorch★ 152
- github.com/hynkis/mi-hponone★ 10
- github.com/hynkis/MIHOnone★ 10
- github.com/billchan226/poar-srl-4-robottf★ 6
- github.com/civisanalytics/civisml-extensionsnone★ 0
- github.com/mle-infrastructure/mle-hyperoptnone★ 0
- github.com/tally0818/HyperBandpytorch★ 0
- github.com/thegaussians/hyperband-for-any-modelnone★ 0
Abstract
Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.