In this paper, structural optimization of a sensor fusion system is newly presented to minimize noise sensitivity, thus maximizing the noise robustness in measuring 6-degree-of-freedom (6-DOF) poses of an object at a long distance with higher precision. The target sensor system, the so-called three-beam detector, is composed of three 1-D laser sensors and a vision camera. During the nonlinear computation, 6-DOF poses can be affected by sensor noises along with sensor configuration. Design parameters such as alignment angles, width, and height of the laser sensors are optimized in the sensor configuration that can minimize an objective function of the 6-DOF pose errors due to the sensor noise. To show sensing performance against the sensor noises, the sensing precision of the optimally configured system is compared with that of the parallel configuration system through numerical simulations and experiments. The proposed process of optimizing sensor configuration can be also applied to other sensor fusion systems to minimize the noise sensitivity.