On the Self-Penalization Phenomenon in Feature Selection

Abstract

We describe an implicit sparsity-inducing mechanism based on minimization over a family of kernels: equation* _ , f~E[L(Y, f( ^1/q X)] + _n \|f\|_H_q^2~~subject to~~ 0, equation* where L is the loss, is coordinate-wise multiplication and H_q is the reproducing kernel Hilbert space based on the kernel k_q(x, x') = h(\|x-x'\|_q^q), where \|\|_q is the _q norm. Using gradient descent to optimize this objective with respect to leads to exactly sparse stationary points with high probability. The sparsity is achieved without using any of the well-known explicit sparsification techniques such as penalization (e.g., _1), early stopping or post-processing (e.g., clipping). As an application, we use this sparsity-inducing mechanism to build algorithms consistent for feature selection.

Tasks

Reproductions