Ridge Regression with Frequent Directions: Statistical and Optimization Perspectives

Abstract

Despite its impressive theory \& practical performance, Frequent Directions (fd) has not been widely adopted for large-scale regression tasks. Prior work has shown randomized sketches (i) perform worse in estimating the covariance matrix of the data than fd; (ii) incur high error when estimating the bias and/or variance on sketched ridge regression. We give the first constant factor relative error bounds on the bias \& variance for sketched ridge regression using fd. We complement these statistical results by showing that fd can be used in the optimization setting through an iterative scheme which yields high-accuracy solutions. This improves on randomized approaches which need to compromise the need for a new sketch every iteration with speed of convergence. In both settings, we also show using Robust Frequent Directions further enhances performance.

Tasks

Reproductions