Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

Abstract

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set A is defined to encompass the majority of commonly used activation functions, such as ReLU, LeakyReLU, ReLU^2, ELU, CELU, SELU, Softplus, GELU, SiLU, Swish, Mish, Sigmoid, Tanh, Arctan, Softsign, dSiLU, and SRS. We demonstrate that for any activation function A, a ReLU network of width N and depth L can be approximated to arbitrary precision by a -activated network of width 3N and depth 2L on any bounded set. This finding enables the extension of most approximation results achieved with ReLU networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,\,depth) scaling factors can be further reduced from (3,2) to (1,1) if falls within a specific subset of A. This subset includes activation functions such as ELU, CELU, SELU, Softplus, GELU, SiLU, Swish, and Mish.

Reproductions