To estimate structural displacement, a visually servoed paired structured light system (ViSP) was proposed in previous studies. The ViSP is composed of two sides facing each other, each with one or two laser pointers, a 2-DOF manipulator, a camera, and a screen. By calculating the positions of the laser beams projected onto the screens and rotation angles of the manipulators, relative 6-DOF displacement between two sides can be estimated. Although the performance of the system has been verified through various simulations and experimental tests, it has a limitation that the accuracy of the displacement measurement depends on the alignment of the laser pointers. In deriving the kinematic equation of the ViSP, the laser pointers were assumed to be installed perfectly normal to the same side screen. In reality, however, this is very difficult to achieve due to installation errors. In other words, the pose of laser pointers should be calibrated carefully before measuring the displacement. To calibrate the initial pose of the laser pointers, a specially designed jig device is made and employed. Experimental tests have been performed to validate the performance of the proposed calibration method and the results show that the estimated displacement with the initial pose calibration increases the accuracy of the 6-DOF displacement estimation.