Neural Dynamic Focused Topic Model

Abstract

Topic models and all their variants analyse text by learning meaningful representations through word co-occurrences. As pointed out by Williamson et al. (2010), such models implicitly assume that the probability of a topic to be active and its proportion within each document are positively correlated. This correlation can be strongly detrimental in the case of documents created over time, simply because recent documents are likely better described by new and hence rare topics. In this work we leverage recent advances in neural variational inference and present an alternative neural approach to the Focused Topic Model and its dynamic extensions. Indeed, we develop a neural model for topic evolution which exploits a compound Bernoulli structure in order to track the appearances of topics, thereby decoupling their activities from their proportions. On three different corpora namely, the UN general debates, the collection of NeurIPS papers, and the ACL Anthology dataset, our model outperforms competing neural variational topic models.

Tasks

Reproductions