A recent trend in vision-based displacement measurement is to place a camera at the measurement point and capture the images of the surrounding areas. In this scheme, a proper region of interest (ROI) should be selected from the captured images. This paper proposes an automated ROI selection technique to improve displacement estimation accuracy. The image frames that capture larger movements of the surrounding areas were selected, and the features in the selected frames were grouped using clustering algorithms. The feature group with consistent movement and high density was finally selected as the optimum ROI. The proposed technique was validated through laboratory and field tests. A displacements estimation technique previously proposed by the authors were used to compared the optimum ROI and four intuitively selected ROIs. In all the tests, the displacement estimates from the optimum ROI showed a smaller RMSE (less than 2 mm) than those from other ROIs.