Bat-inspired temporal audiovisual fusion neural network in adverse weather conditions

Abstract

This paper deals with a neural network that mimics a bat and provides 3D information by tracking a moving object even in severe weather such as sudden fog or rain. In the case of an ultrasonic sensor, it is possible to obtain the location of an object because it is robust even in bad weather, but it is impossible to accurately predict the size of the object. In contrast, a visual sensor such as a camera can obtain the location and size of an object, but has a disadvantage in that it does not operate properly in harsh environment. We implemented a network that can provide 3D information of objects even in bad weather conditions by mapping information obtained from images and ultrasound. We demonstrated the performance of the network through intersection over union (IoU) values, and these experimental results showed that objects can be tracked even in severe weather through the mutual complement of ultrasound and camera sensors.