Unpicking Data at the Seams: VAEs, Disentanglement and Independent Components

Abstract

Disentanglement, or identifying salient statistically independent factors of the data, is of interest in many areas of machine learning and statistics, such as synthetic data generation with controlled properties, robust classification of features, parsimonious encoding, and improving our understanding of the generative process underlying the data. Disentanglement is observed in several generative paradigms, including Variational Autoencoders (VAEs), Generative Adversarial Networks and diffusion models. Particular progress has recently been made in understanding disentanglement in VAEs, where the choice of diagonal posterior covariance matrices is proposed to promote mutual orthogonality between columns of the decoder's Jacobian. We continue this thread to show how such linear independence translates to statistical independence, completing the chain in understanding how the VAE's objective identifies independent components of, or disentangles, the data.

Tasks

Reproductions