Robust localization under appearance changes with multi-modal sensor fusion

Abstract

In this dissertation, we propose a multi modal sensor fusion for robust SLAM under changing environments. Multimodal sensor fusion includes both the interoperability of various types of vision sensors and the integration between vision sensors and range sensors. In this thesis, we propose a deep learning-based multi modal sensor fusion as well as the place recognition and SLAM framework based on multi modal fusion. The framework includes the methods to obtain spatial and location information through event, thermal imaging, optical cameras, and LiDAR sensors, as well as the intercompatibility and possible use cases of selected sensor sets. These fusion methods opens a way to solve data scarcity problem of robotic sensors in the varying real world

event cameras on drone maneuvering at high speeds or a LiDARs on the high fidelity mobile mapping systems, with integration from cross-modal database. In the first part of this dissertation, we discuss about the place recognition based on various types of data or sensors

e.g. event, language, depth image, respectively. A place recognition is crucial for accurate localization of robots, especially for vision-based systems that may affect from appearance changes. In a successful case, errors from the odometry module are reduced by correcting pose information from known values in the database, and also the map is maintained by the correspondences obtained between current measurements and the database. EventVLAD presents a method for robust place recognition over appearance changes, using an event camera. The module efficiently removes noise generated in the wide dynamic detection range of the event sensor and extracts the location descriptor, showing higher accuracy for lighting condition changes, than the existing visual place recognition algorithm. NRF-VPR shows that the feature space of the learned image-language model can be used as an image descriptor for place recognition, demonstrating that explanatory AI in place recognition is feasible. LC2 shows the method of depth-based global localization, that multi modal sensor fusion of camera-LiDAR can be achieved through depth image conversion. In the second part of the dissertation, we propose a method to improve the accuracy of pose estimation and create a map by assisting information of various modalities. This includes a use cases, such as heat mapping with thermal camera and a 2D range scanner, or matching camera images to globally correct the pose estimation from event cameras or LiDARs. Event cameras and LiDARs provide spatial and temporal sophistication, so they are widely used in SLAM. However, the databases of sensors are not widely constructed and global correction is often unavailable. We propose that this data scarcity problem can be solved through fusion with image databases, that has been already captured under the ideal lighting conditions. To achieve this, we propose to fuse each sensor into an image database and globally calibrate the pose from correspondences obtained from EventVLAD and LC2 suggested in the first part. This fusion technique, finally proposed in this thesis, is provided as a modulized open source. The last part of the dissertation is completed by analyzing the contribution and limitations of the robust place recognition.