f-Divergence Thermodynamic Variational Objective: a Deformed Geometry Perspective

Abstract

In this paper, we propose a f-divergence Thermodynamic Variational Objective (f-TVO). f-TVO generalizes the Thermodynamic Variational Objective (TVO) by replacing Kullback–Leibler (KL) divergence with arbitary differeitiable f-divergence. In particular, f-TVO approximates dual function of model evidence f^*(p(x)) rather than the log model evidence p(x) in TVO. f-TVO is derived from a deformed -geometry perspective. By defining -exponential family exponential, we are able to integral f-TVO along the -path, which is the deformed geodesic between variational posterior distribution and true posterior distribution. Optimizing scheme of f-TVO includes reparameterization trick and Monte Carlo approximation. Experiments on VAE and Bayesian neural network show that the proposed f-TVO performs better than cooresponding baseline f-divergence variational inference.

Tasks

Reproductions