This paper proposes a large-scale graph-based SLAM (Simultaneous Localization and Mapping) approach with the traffic sign data involved in the HD (high definition) map. The graph is structured by the IMU factor, LiDAR-inertial factor, traffic sign factor, and loop closure factor. The IMU preintegration generates the IMU factor. The IMU pre-integration result is used to de-skew point cloud in the preprocessing and is used for LiDAR odometry optimization as an initial guess. The traffic sign factor is generated by the detection and map matching process. The loop closure is searched based on the geometry information in Euclidean space. The graph structure is optimized when the loop closure factor or the traffic sign factor is updated. The proposed method solves the long-term drift error problem of the SLAM in the large-scale environment and also improves the localization accuracy compare with the state-of-the-art LiDAR-inertial odometry methods. Also, the proposed method is intensively tested with collected datasets in the city where the GPS multi-path problem occurs and inside the campus.