An Approximation Algorithm for Optimal Subarchitecture Extraction

Code

• github.com/alexa/bort
  mxnet★ 470
• github.com/kmz4/QHACK2021
  none★ 0

Abstract

We consider the problem of finding the set of architectural parameters for a chosen deep neural network which is optimal under three metrics: parameter size, inference speed, and error rate. In this paper we state the problem formally, and present an approximation algorithm that, for a large subset of instances behaves like an FPTAS with an approximation error of |1- |, and that runs in O(|| + |W^*_T|(1 + |||B|||/(\, s^3/2))) steps, where and s are input parameters; |B| is the batch size; |W^*_T| denotes the cardinality of the largest weight set assignment; and || and || are the cardinalities of the candidate architecture and hyperparameter spaces, respectively.

Reproductions