IKUN: Initialization to Keep snn training and generalization great with sUrrogate-stable variaNce

Code

• github.com/maechd/surrogatevarstabe
  OfficialIn paper★ 0

Abstract

Weight initialization significantly impacts the convergence and performance of neural networks. While traditional methods like Xavier and Kaiming initialization are widely used, they often fall short for spiking neural networks (SNNs), which have distinct requirements compared to artificial neural networks (ANNs). To address this, we introduce IKUN, a variance-stabilizing initialization method integrated with surrogate gradient functions, specifically designed for SNNs. IKUN stabilizes signal propagation, accelerates convergence, and enhances generalization. Experiments show IKUN improves training efficiency by up to 50\%, achieving 95\% training accuracy and 91\% generalization accuracy. Hessian analysis reveals that IKUN-trained models converge to flatter minima, characterized by Hessian eigenvalues near zero on the positive side, promoting better generalization. The method is open-sourced for further exploration: https://github.com/MaeChd/SurrogateVarStabe.

Reproductions