Precise Performance of Linear Denoisers in the Proportional Regime

Abstract

In the present paper we study the performance of linear denoisers for noisy data of the form x + z, where x R^d is the desired data with zero mean and unknown covariance Σ, and z N(0, Σ_z) is additive noise. Since the covariance Σ is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples x_1,,x_n R^d from the true distribution. A standard approach would then be to estimate Σ from the samples and use it to construct an ``empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser W from the data itself. In particular, we synthetically construct noisy samples x_i of the data by injecting the samples with Gaussian noise with covariance Σ_1 Σ_z and find the best W that approximates Wx_i x_i in a least-squares sense. In the proportional regime nd κ> 1 we use the Convex Gaussian Min-Max Theorem (CGMT) to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over Σ_1 to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the ``empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as κ.

Reproductions