Deep neural network with multimodal auxiliary tasks for visual question answering

Abstract

In this thesis, we have proposed a new training method which adds auxiliary tasks to existing models. For visual question answering problem, it is important to increase the mutual information among questions, images, and answers. By reconstructing features of the training data from answer, we were able to guide learning process more efficiently. The proposed method is not limited to a specific model, and it can also improve the performance of model while preserving the size of models.