Multi-modal depth estimation from misaligned thermal and RGB images

Abstract

Depth estimation is a research area that focuses on predicting the depth of each pixel in an input image when matched to a 3D space. Research on depth estimation is highly applicable in various fields such as autonomous driving and virtual reality. Particularly in recent years, it has become a crucial study in the field of autonomous driving and robot vision. In this thesis, we propose an effective deep learning method for estimating the depth of images by simultaneously utilizing thermal and RGB images, which are actively studied for enhancing driver and pedestrian safety through automatic pedestrian detection in autonomous driving. In this thesis, we propose the method that complementarily predicts the depth from misaligned thermal and RGB images. Specifically, to utilize consistent information from thermal images captured during nighttime and RGB images representing consistent information during daytime, we propose: (i) feature extraction from misaligned thermal and RGB images and their Cross-fusion module, (ii) a shared encoder and decoder structure for multi-modal image input, and (iii) Multi-objective training strategy for simultaneous supervised training from multi-modal supervision. In particular, we use cross-attention methods to effectively extract features for depth prediction from corresponding positions in misaligned thermal and RGB images. Through various experiments, our proposed method demonstrates its effectiveness, achieving performance improvements of 7%-points and 4%-points, respectively, compared to using only each modal input (thermal or RGB images).