Object Tracking is a technique for tracking moving objects over time in a video image. Using object tracking technique, many research are conducted such a detecting dangerous situation and recognizing the movement of nearby objects in a smart car. However, it still remains a challenging task such as occlusion, deformation, background clutter, illumination variation, etc. In this paper, we propose a novel deep visual object tracking method that can be operated in robust to many challenging task. For the robust visual object tracking, we proposed a Convolutional Neural Network(CNN) which shares weight of the convolutional layers. Input of the CNN is a three; first frame object image, object image in a previous frame, and current search frame containing the object movement. Also we propose a method to consider the motion of the object when determining the current search area to search for the location of the object. Extensive experimental results on a authorized resource database showed that the proposed method outperformed than the conventional methods.