The information bottleneck method
Naftali Tishby, Fernando C. Pereira, William Bialek
Code Available — Be the first to reproduce this paper.
ReproduceCode
Abstract
We define the relevant information in a signal x X as being the information that this signal provides about another signal y . Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal x requires more than just predicting y, it also requires specifying which features of play a role in the prediction. We formalize this problem as that of finding a short code for that preserves the maximum information about . That is, we squeeze the information that provides about through a `bottleneck' formed by a limited set of codewords . This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure d(x,) emerges from the joint statistics of and . This approach yields an exact set of self consistent equations for the coding rules X and . Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.