Distributed online learning for topic models

Abstract

A major obstacle in using a probabilistic topic model, such as Latent Dirichlet Allocation (LDA) or Hierarchical Dirichlet Processes (HDP) is the amount of time it takes for posterior inference, especially for Web data which are huge and continuously expanding. Recent developments in distributed inference algorithms and minibatch-based online learning algorithms have offered partial solutions for this problem. In this paper, I propose a distributed online learning algorithm for LDA and HDP for dealing with both aspects of this problem at once. I apply our learning algorithm to three datasets: a corpus of 973K Twitter conversations and 4.8M Wikipedia articles used for a quantitative evaluation of our algorithm, and a larger corpus of 5.1M Twitter conversations for a case study. I compare our algorithm with the distributed version of variational inference using MapReduce and online learning using stochastic variational inference. I show that our learning algorithm achieves the same model fit and topic quality as the other inference algorithms but within a much shorter learning time. I conduct a case study using our distributed online learning framework to visualize how the topic proportions change over time in a stream of Web documents. Through this case study, I discover interesting temporal dynamics of topics in Twitter conversations.