Dynamic ReLU
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dong-Dong Chen, Lu Yuan, Zicheng Liu
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/Islanna/DynamicReLUpytorch★ 205
- github.com/Coldestadam/DynamicHeadpytorch★ 30
- github.com/MindCode-4/code-11/tree/main/dynamic-relumindspore★ 0
- github.com/reeered/Dynamic-ReLUmindspore★ 0
- github.com/MindCode-4/code-6/tree/main/dynamic-relumindspore★ 0
Abstract
Rectified linear units (ReLU) are commonly used in deep neural networks. So far ReLU and its generalizations (non-parametric or parametric) are static, performing identically for all input samples. In this paper, we propose dynamic ReLU (DY-ReLU), a dynamic rectifier of which parameters are generated by a hyper function over all in-put elements. The key insight is that DY-ReLU encodes the global context into the hyper function, and adapts the piecewise linear activation function accordingly. Compared to its static counterpart, DY-ReLU has negligible extra computational cost, but significantly more representation capability, especially for light-weight neural networks. By simply using DY-ReLU for MobileNetV2, the top-1 accuracy on ImageNet classification is boosted from 72.0% to 76.2% with only 5% additional FLOPs.