This paper considers a scene graph generation algorithm referred to as Relational Object Message Passing to enhance the representation of object relationships. In the scene graph, there are various kinds of relationship pattern depending on both subject and object. For example, given an image of a person and a bicycle, the relationship can be interpreted as the man riding, pushing, leaning against, or even falling off the bicycle. In this paper, for acquiring the good relational representation, two types of message passing algorithms are investigated: it operates on direct graph and undirect graph. The experiments showed the direct graph-based message passing is better than the undirect one, while comparing with the baseline.