Multidimensional mining of search logs based on topic-concept cube approach

Abstract

In addition to search queries and the corresponding click-through information, search engine logs record multidimensional information about user search activities, such as search time, location, vertical, and search device. Multidimensional mining of search logs can provide novel insights and useful knowledge for both search engine users and developers. How can we develop a search engine service to support multidimensional mining of search logs effectively and efficiently? In this paper, we describe our topic-concept cube project which addresses the business need and answers several challenges. First, to semantically summarize a set of search queries and click-through data, we develop a novel topic-concept model which learns a hierarchy of concepts and topics automatically from search logs. Second, to handle a huge amount of log data, we develop distributed algorithms for learning model parameters efficiently. Third, we present alternative approaches for computing a topic-concept cube which supports multidimensional mining of search log data online. Last, we report an empirical study verifying the effectiveness and efficiency of our approach on a real data set of 1.96 billion queries and 2.73 billion clicks.