Deep learning based approaches for multimodal video question answering

Abstract

This dissertation considers the problem of Multimodal Video Question Answering (MVQA) which aims at joint understanding of video and accompanied subtitles to answer the given question. Compared to visual question answering (VQA) which is question answering on a single image, MVQA is challenging in two aspects: (1) it requires pinpointing the temporal parts relevant to answer the question as input is long untrimmed video, and (2) it involves reasoning on heterogeneous modality where different question requires different modality to answer the question. We propose two MVQA networks to address aforementioned challenges: (1) Progressive Attention Memory Network (PAMN), and (2) Modality Shifting Attention Network (MSAN). Experimental results on MovieQA and TVQA shows proposed PAMN and MSAN achieves significant performance improvement compared to previous state-of-the-art methods. Furthermore, we propose Structured Co-reference Graph Attention for Video-grounded Dialog (VideoDial) task and showed performance boost on AVSD benchmark.