Learning deep representations by mutual information estimation and maximization
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/rdevon/DIMOfficialIn paperpytorch★ 0
- github.com/schzhu/learning-adversarially-robust-representationspytorch★ 23
- github.com/HolenYHR/Deepinfo_pytorchpytorch★ 0
- github.com/DuaneNielsen/DeepInfomaxPytorchpytorch★ 0
- github.com/bojone/infomaxtf★ 0
- github.com/ifding/simple-Infomax-pytorchpytorch★ 0
- github.com/createamind/DIM_Commentedpytorch★ 0
- github.com/jqhoogland/rgpytf★ 0
Abstract
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.