Co-burst based topical word extraction for text summarization and search

Abstract

The rapid increase of electronical text necessitates reliable automatic text analysis that supports human knowledge tasks. Among the most promising topics are document summarization and topical search for desired documents. This research suggests considering the writer’s cognitive process of communication, formed as a sequence of topics. A topical word set in a part of a document may indicate the schemas that compose the subject in that point of discourse. If the topical word sets can be extracted, they can be used to improve both text summarization and document search. The co-burst based topical word extraction method is proposed to find the schematic terms. Burst analysis detects where a word is more active or bursty than in the other parts. A set of words having bursts together, or co-burst, may represent the topic schemas in that part. The proposed method is implemented and applied first to the single-document summarization. A knowledge-based approach using a knowledge base is used as a complementary method. The result shows that the new approach outperforms the current state-of-the-art summarization, verifying that using the schema terms found by the co-burst detection method has a great effect. The approach is also applied to multi-document summarization. The problem is formulated as a multi-objective optimization with two objective functions: coverage and diversity. For the coverage objective function, k-means clustering, knowledge-based, and co-burst-based topical coverage functions are examined. The results are again better than the conventional methods and show greater robustness in realistic situations where the word sequence and order varied. Since a topic schema can be represented by a word set, extracting topics will benefit text-based works such as searching for documents or sections of topical interest. The idea is presented by an exemplar topical search system that visualizes the topics and allows the user to interactively find relevant documents.