Multi-view full sentence visual question answering with full sentence answer network and question-driven object attention network

Abstract

Visual Question Answering (VQA) is a task which answers a question about a given image. So, a model for VQA needs understanding of images and questions, and reasoning method based on a given image and question. Previous researches on VQA are mainly focused on better reasoning and understanding of images or questions. However, in a real VQA application where a robot or a mobile device interacts with human, the VQA model should handle surrounding environment rather than an image taken at specific time. We name the task as Multi-view VQA (MV-VQA) when the object of the task is to get a word answer, and Multi-view Full Sentence VQA (MV-FSVQA) when the object of the task is to get a full sentence answer. We propose a question-driven object-based attention model for the tasks. Furthermore, we separately train a seq2seq model for FSVQA and MV-FSVQA task to get better full sentence answer unlike the baseline algorithm. We carried out various experiments on VQA, FSVQA, MV-VQA, and MV-FSVQA with MS COCO dataset and customized datasets. We show that our model achieves improvements over the baseline especially in Multi-view scenarios and demonstrate the feasibility of the proposed model for real application.