Previous digital image forensics focused on the low-level features that include traces of the image modifying history. In this paper, we present a framework to detect the manipulation of images through a contextual violation. First, we proposed a context learning convolutional neural networks (CL-CNN) that detects the contextual violation in the image. In combination with a well-known object detector such as R-CNN, the proposed method can evaluate the contextual scores according to the combination of objects in the image. Through experiments, we showed that our method effectively detects the contextual violation in the target image.