Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension

Abstract

Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension d_0 scales at least logarithmically in the number of samples n. In this work we remove both of these requirements and instead provide bounds in terms of a measure of the collinearity of the data: notably these bounds hold with high probability even when d_0 is held constant versus n. We prove our results through a novel application of the hemisphere transform.

Tasks

Reproductions