Scene graph generation with attended predicate prediction

Abstract

Scene graph generation is the task of detecting objects and their pairwise visual relationships in an image. This task is challenging because it requires not only detecting object instances of various categories but also high-level understandings of visual relation. Besides accurate object detection, a high-performance predicate classifier is critical for an applicable scene graph generation model for real-world systems. To the best of our knowledge, all prior works in scene graph generation align a union region of two objects and flatten it into a vector without any processing in the middle. In this work, we propose a Union Region Attention module to focus on objects and the interaction between them in union image regions. Namely, this module generates a spatial attention mask for the union region of two objects by using embedding features from their labels, appearances, and spatial configuration. The mask puts higher weights on important points of objects than irrelevant points in the union region. Despite its simplicity, the model with our proposed module outperforms other state-of-the-art predicate prediction methods. Experiments on Visual Genome confirm the efficacy of our module. We also conduct ablation studies to provide insight into the operation of each component of our union region attention module.