Convergence of Shallow ReLU Networks on Weakly Interacting Data

Code

• github.com/leodana2000/Convergence-High-Dimension
  Officialpytorch★ 0

Abstract

We analyse the convergence of one-hidden-layer ReLU networks trained by gradient flow on n data points. Our main contribution leverages the high dimensionality of the ambient space, which implies low correlation of the input samples, to demonstrate that a network with width of order (n) neurons suffices for global convergence with high probability. Our analysis uses a Polyak- ojasiewicz viewpoint along the gradient-flow trajectory, which provides an exponential rate of convergence of 1n. When the data are exactly orthogonal, we give further refined characterizations of the convergence speed, proving its asymptotic behavior lies between the orders 1n and 1n, and exhibiting a phase-transition phenomenon in the convergence rate, during which it evolves from the lower bound to the upper, and in a relative time of order 1(n).

Reproductions