Topic models for discovering latent inter-dependency within text data

Abstract

Topic models refer to a group of bayesian statistical models that discover latent structure of documents inthe form of human-readable topics. They have been widely researched and applied in industry for tasks such as document classification, sentiment analysis, and review prediction. Recently we have witnessed two cases of major explosion in text data as well as a new trend in statistical language models. These changes call for more advanced models. The goal of this thesis is to present novel Bayesian generativemodels that are capable of gaining deeper understanding from the data. First, most of human knowledgeand information have become digitized with efforts such as Wikipedia and Gutenberg project. Oftentopic model is used to extract topics and obtain insights on the corpus, yet it is much harder to learnrelations and structure among topics. I build recursive Chinese Restaurant Process (rCRP), a novelnonparametric generative model that discovers hierarchical structure of topics. Second, traces of ourdaily lives are transferred online with proliferation of mobile devices. This gave rise to coupling of textdata with other types of data such as text-photo (Facebook, Instagram), text-video (Youtube), andtext-review score (Amazon). With its flexibility, researchers have attempted to extend topic model fromits original form of LDA to account for additional data. However, text-click data, equivalent to mostonline user activity, has not been researched. I build Headline Click-based Topic Model (HCTM), a novelgenerative model that learns click-value of words for relevant semantic context. Finally, the developmentof competing models such as word embedding has precipitated the need for modeling specific contextsof individual words. Topic model is good at extracting general topics of an entire document, however itrequires better modeling of local context at individual word locations. Topic model is good at extractinggeneral topics of an entire document, however word embedding excels at extracting local context atspecific position of document. I build Dual Context Topic Model (DCTM), a novel generative modelthat accounts for both document context and local context of individual words.