In visual object tracking, exploiting correlation filters to track the target of interest has been flourished, and, by adopting a circulant form of an image or feature descriptors jointed with the convolution theorem, these correlation filter trackers surpass many of the previous state-of-the-art trackers in both tracking speed and stability. Nevertheless, when the appearance of the target object abruptly changes due to occlusion, background cluttering, or viewpoint variation, even the aforementioned correlation filter trackers still tend to fail to compute a reliable correlation output. Concerned with this problem, we propose a method that observes the locational drift of the correlation peak from the desired location. Utilizing this information, we restrict the searching range of the correlation peak to increase the accuracy of the tracker. We verify the performance of the proposed tracker by using 2014 Visual Object Tracking Challenge benchmark dataset.