Moment proposal network for multi-modal video question answering

Abstract

This paper proposes Moment Proposal Network (MPN) for Multimodal Video Question Answering (MMVQA). MMVQA requires the understanding of the story of TV shows based on the video and conversation to answer the given question. Existing methods rely on temporal attention mechanisms to retrieve the relevant moment to the current QA. However, there are two main limitations. One is that the attention map tends to be blurred as the video length is increased which hinders pinpointing the required moment. Another is that the contribution of each modality is not considered. To this end, Moment Proposal Network (MPN) is proposed to retrieve the golden moment by a hard attention mechanism which reduces the search space for the subsequent reasoning networks. In addition, MPN can dynamically determine the importance of each modality for the given question by Modality Importance Modulation. MPN is trained to solve a ranking problem between the query and the candidate moment proposals. The experiments on publicly available dataset TVQA show that MPN achieves state-of-the-art performance and provides interpretability of where to attend.