3D vehicle localization using camera system

Abstract

3D vehicle detection is one of the fundamental task for the autonomous driving. Based on the detection results in the dynamic road environment, path planning and trajectory planning can be established to control the vehicle. LiDAR is one of the most promising sensor for the task of 3D vehicle detection, it shows the state-of-the-art performance in the target task. However, there are deficits of this unique sensor, such as high expenditure and calibration issue such that the necessity of the camera-based approach is still in-need. The goal of this thesis is to address the fundamental problem of the camera-based 3D vehicle localization methods and overcome the difficulty to achieve the highly accurate performance. High-quality depth information is required to perform 3D vehicle detection, consequently, there exists a large performance gap between camera and LiDAR-based approaches.

Our monocular camera-based 3D vehicle localization method alleviates the dependency on high-quality depth maps by taking advantage of the commonly accepted assumption that the observed vehicles lie on the road surface. We propose a two-stage approach that consists of a segment network and a regression network, called Segment2Regress. For a given single RGB image and a prior 2D object detection bounding box, the two stages are as follows: 1) The segment network activates the pixels under the vehicle (modeled as four line segments and a quadrilateral representing the area beneath the vehicle projected on the image coordinate). These segments are trained to lie on the road plane such that our network does not require full depth estimation. Instead, the depth is directly approximated from the known ground plane parameters. 2) The regression network takes the segments fused with the plane depth to predict the 3D location of a car at the ground level. To stabilize the regression, we introduce a coupling loss that enforces structural constraints. The efficiency, accuracy, and robustness of the proposed technique are highlighted through a series of experiments and ablation assessments. These tests are conducted on the KITTI bird's eye view dataset where Segment2Regress demonstrates state-of-the-art performance.

Stereo vision system is widely utilized to infer the depth information. Depth is essential measurements that not only capture the 3-dim road environment, but also include the shape of the 3D vehicle in the dynamic road environment. In this thesis, we introduce the structure-aware depth prediction method for autonomous driving. To impose the structural prior into the disparity, we introduce the 3-dim region proposal network that captures the location of the vehicles with referential camera viewpoint. Comparing with the previous method, our method alleviate the flying-point-cloud problem since structure-aware loss reduce the uncertainty nearby the edges of the vehicles. Finally, our method achieves the state-of-the-art performance in the KITTI[11] Stereo benchmark.

Through the thesis, we address the depth dependency problem in the monocular-based approaches and the disparity-uncertainty problem in the stereo-based methods. In the first method, we utilize the plane assumptions and the plane depth, and introduce the fusion-by-normalization and the coupling loss to decrease the sensitivity to the quality of the monocular scene depth. In the second method, we exploit the structure of vehicles to minimize the disparity-uncertainty nearby the edges of vehicles. In conclusion, this effort is to continue the necessity and the validity of the camera-based approaches for autonomous driving.