S-System, Geometry, Learning, and Optimization: A Theory of Neural Networks

Abstract

We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Particularly, we present an algorithm framework, Hierarchical Measure Group and Approximate System (HMGAS), nicknamed S-System, of which NNs are special cases. In addition to many other results, the framework enables us to prove that 1) NNs implement renormalization group (RG) using information geometry, which points out that the large scale property to renormalize is dual Bregman divergence and completes the analog between NNs and RG; 2) and under a set of realistic boundedness and diversity conditions, for large size nonlinear deep NNs with a class of losses, including the hinge loss, all local minima are global minima with zero loss errors, using random matrix theory.

Tasks

Reproductions