In this paper, we present an underwater visual mapping method for the three-dimensional (3D) reconstruction of a moderately curved ship hull surface using a stereo vision system. The proposed approach estimates the local hull surface by extracting 3D point clouds from stereo image pairs, and the relative poses between the pairs are calculated by matching the corresponding point cloud points. A surfel model is extracted from each point cloud set by fitting a plane model, and a smoothness factor is applied between nearby surfels for a smooth 3D surface reconstruction. The camera trajectory and the surfel map are optimized through the graph-based simultaneous localization and mapping (SLAM) framework. A 3D surface mesh is generated with the optimized surfel poses, and the corresponding images are projected into the surface plane, texturing the surface. The performance of the proposed approach is shown with experimental data obtained from an actual ship hull inspection.