Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance

Code

• github.com/ictseoyoungmin/swish-t-pytorch
  OfficialIn paperpytorch★ 3
• github.com/pwc-1/Paper-9/tree/main/3/swish-t
  mindspore★ 0
• github.com/MindCode-4/code-9/tree/main/swish-t
  mindspore★ 0
• github.com/pwc-1/Paper-9/tree/main/4/swish-t
  mindspore★ 0
• github.com/reeered/Swish-T-MindSpore
  mindspore★ 0

Abstract

We propose the Swish-T family, an enhancement of the existing non-monotonic activation function Swish. Swish-T is defined by adding a Tanh bias to the original Swish function. This modification creates a family of Swish-T variants, each designed to excel in different tasks, showcasing specific advantages depending on the application context. The Tanh bias allows for broader acceptance of negative values during initial training stages, offering a smoother non-monotonic curve than the original Swish. We ultimately propose the Swish-T_C function, while Swish-T and Swish-T_B, byproducts of Swish-T_C, also demonstrate satisfactory performance. Furthermore, our ablation study shows that using Swish-T_C as a non-parametric function can still achieve high performance. The superiority of the Swish-T family has been empirically demonstrated across various models and benchmark datasets, including MNIST, Fashion MNIST, SVHN, CIFAR-10, and CIFAR-100. The code is publicly available at https://github.com/ictseoyoungmin/Swish-T-pytorch.

Reproductions