Nowadays researchers have been trying to recognize human activity using human joint information in RGB-D human activity dataset. However, it is necessary to recognize objects interacting with a human to improve human activity classification accuracy because an object is able to give us an important information to classify human activity. In this paper, we describe the effective way of detecting object region related to human activity. We used human joint information and its movement properties. The experimental result shows that object region interacting with a human is detected well by our proposed method.