Independent projections of diffusions: Gradient flows for variational inference and optimal mean field approximations

Abstract

What is the optimal way to approximate a high-dimensional diffusion process by one in which the coordinates are independent? This paper presents a construction, called the independent projection, which is optimal for two natural criteria. First, when the original diffusion is reversible with invariant measure _*, the independent projection serves as the Wasserstein gradient flow for the relative entropy H(\,|\,_*) constrained to the space of product measures. This is related to recent Langevin-based sampling schemes proposed in the statistical literature on mean field variational inference. In addition, we provide both qualitative and quantitative results on the long-time convergence of the independent projection, with quantitative results in the log-concave case derived via a new variant of the logarithmic Sobolev inequality. Second, among all processes with independent coordinates, the independent projection is shown to exhibit the slowest growth rate of path-space entropy relative to the original diffusion. This sheds new light on the classical McKean-Vlasov equation and recent variants proposed for non-exchangeable systems, which can be viewed as special cases of the independent projection.

Tasks

Reproductions