Visual reasoning with neural program induction

Abstract

In this dissertation, I propose a neural program induction method for learning visual reasoning task. First, I point out the reason why the existing deep learning methods do not generalize to new types of problems that have not been seen during training. Specifically in PGM(Procedurally Generated Matrices), a visual reasoning benchmark that can measure generalization ability toward novel composition of concepts, I show that a deep learning model without sufficient prior knowledge of PGM domain always can fail, even though the domain consists of very basic concepts. As an alternative solution to this problem, I propose a neural-symbolic system to solve visual reasoning problems. Firstly, I define a domain-specific language(DSL) consisting of functions designed to solve PGM problems. Next, a probability distribution over DSL sentences, or programs, is modeled with train samples and neural networks. Finally, based on the predicted probability distribution, the program is executed sequentially from the highest probability until one correct answer is found among the eight options. The proposed method achieves an accuracy of 97.53% with an average of 9.12 program executions per problem in PGM held-out triples test set, where the novel composition of objects, attributes and relations appears.