Kernel Distillation for Fast Gaussian Processes Prediction

Abstract

Gaussian processes (GPs) are flexible models that can capture complex structure in large-scale dataset due to their non-parametric nature. However, the usage of GPs in real-world application is limited due to their high computational cost at inference time. In this paper, we introduce a new framework, kernel distillation, to approximate a fully trained teacher GP model with kernel matrix of size n n for n training points. We combine inducing points method with sparse low-rank approximation in the distillation procedure. The distilled student GP model only costs O(m^2) storage for m inducing points where m n and improves the inference time complexity. We demonstrate empirically that kernel distillation provides better trade-off between the prediction time and the test performance compared to the alternatives.

Tasks

Reproductions