Investigating Generalization by Controlling Normalized Margin

Code

• github.com/alexfarhang/margin
  OfficialIn paperpytorch★ 2

Abstract

Weight norm \|w\| and margin participate in learning theory via the normalized margin /\|w\|. Since standard neural net optimizers do not control normalized margin, it is hard to test whether this quantity causally relates to generalization. This paper designs a series of experimental studies that explicitly control normalized margin and thereby tackle two central questions. First: does normalized margin always have a causal effect on generalization? The paper finds that no -- networks can be produced where normalized margin has seemingly no relationship with generalization, counter to the theory of Bartlett et al. (2017). Second: does normalized margin ever have a causal effect on generalization? The paper finds that yes -- in a standard training setup, test performance closely tracks normalized margin. The paper suggests a Gaussian process model as a promising explanation for this behavior.

Tasks

Reproductions