Network fusion based video summarization using visual-semantic features

Abstract

This paper proposes a video summarization method based on network fusion. The goal of this method is to create a meaningful video summary consisting of representative scenes without duplication by using visual and semantic features. To achieve this goal, a final summary is generated by considering the visual and semantic similarity among shots. More specifically, our method uses Convolutional neural networks (CNNs) to extract visual and semantic features. Visual features are the image features from the top layer of CNNs. Semantic features are the word vectors represented by the Word2Vec descriptor. A number of key frames are generated by a shot segmentation and used as input of the CNNs. A visual or semantic network is then constructed by computing the cosine similarity among visual or semantic features. After the similarity is computed, two networks are combined into a fused network by the network fusion process. An optimal video summary is then computed using spectral clustering on the fused network. The performance of this method is evaluated on two datasets, and the results show that it achieves better performance than state-of-the-art video summarization methods.