Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap

Code

• github.com/yulonghui/Uniformity-Gap
  Officialpytorch★ 4
• github.com/MindCode-4/code-7/tree/main/generalizing-and-decoupling-neural-collapse
  mindspore★ 0
• github.com/MindCode-4/code-12/tree/main/generalizing-and-decoupling-neural-collapse
  mindspore★ 0
• github.com/guluguluhhhh/contrib/tree/43-2/application/generalizing-and-decoupling-neural-collapse
  mindspore★ 0
• github.com/MindSpore-scientific-2/code-12/tree/main/generalizing-and-decoupling-neural-collapse
  mindspore★ 0

Abstract

The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown that both cross-entropy loss and mean square error can provably lead to NC. We remove NC's key assumption on the feature dimension and the number of classes, and then present a generalized neural collapse (GNC) hypothesis that effectively subsumes the original NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose a general objective -- hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges to GNC, but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.

Reproductions