Learning based adaptive visual sensor fusion for robust pedestrian detection

Abstract

With the great success of deep neural network, machine learning and computer vision technologies, the era of autonomous driving which is the most promising application of intelligent systems has been advanced. For safety-critical application such as autonomous driving, robustness to rare/unexpected cases is an important issue in practice. One of the promising way for this issue is to use multiple sensors since heterogeneous sensors observe a scene in a different perspective so that the possibility to get useful information to perceive the world could be increased. However, the uniqueness causes difficulty to integrate the information properly. In this dissertation, we cover the whole process of heterogeneous visual sensor fusion from building a multi-sensor system to perception algorithm. The contributions of this dissertation are as follows.

First, we use color images and 3D points from LiDAR since the correspondence could be calculated by sensor calibration. We propose an efficient framework for multiple object detection. For real-time performance, we effectively utilize both 2D/3D information at object proposal stage consisting of removing out-of-interest regions, clustering, and proposal generation steps. Then we apply modality-specific algorithms to each modality to consider both uniqueness and correlation between heterogeneous information. Our method improves the overall detection performance by successfully detecting challenging cases which might be difficult by using a single modality.

Second, we use heterogeneous cameras to capture different wavelength of light. We propose a solution to solve the correspondence/alignment of heterogeneous images by a beam-splitter based hardware system. Using this system, we get aligned image pairs easily and show that the pedestrian detection performance could be significantly improved at day and night. We analyze the usefulness of the aligned thermal information in terms of discriminative power and propose the multispectral ACF(aggregated channel features) to improve the detection performance. We make our benchmark publicly available to encourage the researches for robust pedestrian detection. Our benchmark offers an opportunity for many novel tasks from learning a better representation for deep neural network to a novel cross-modality transformation, i.e. color-to-thermal and thermal-to-color transformation.

Finally, we propose a novel module for deep neural network so that we achieve the performance improvement in normal imaging condition and alleviate performance degradation on abnormal image conditions. Our key idea which is somewhat similar to the philosophy of transfer learning is to change a small number of parameters in the trained neural network. Despite the fixation of trained weights in most of neural networks, we make the fusion parameters, i.e. a small number of parameters in our network, changeable conditioned on the input images. In other words, we propose a scene-adaptive fusion module to inject a flexible characteristic to our networks which predict kernel and bias for convolution layer based on the correlation between low-level features from both the modalities. We show that the predicted parameters from our module form a semantic distribution depending on the current input imaging quality. Our method is trained without seeing failure cases explicitly and achieves the performance improvement from our strong baseline method that is competitive performance with state-of-the-art methods in various normal/abnormal imaging conditions.