Semantic scene recognition in adverse visual conditions for autonomous systems

Abstract

The intelligent mobile robot is expected to provide convenience and improve the quality of human life by performing specific tasks with users based on information obtained by recognizing the surrounding environment. To this end, intelligent mobile robots first need the ability to recognize the surroundings and autonomously drive to the desired destination. To safely drive to the desired destination without collisions, the following functionalities are required. A mobile robot must 1) be able to recognize the type of objects placed around it, 2) predict the distance to the object, and 3) recognize unexpected obstacles that are not used during training. In addition, 4) it should be able to perform the preceding requirements in real-time based on the fast operation speed. To solve these problems, in this paper, we propose a network that simultaneously perform multiple tasks such as semantic segmentation, stereo disparity estimation, and obstacle detection at high computational speed. In particular, when operating a mobile robot in an outdoor environment, it is desirable to detect unexpected road hazards reliably in real-time, especially under varying adverse conditions (e.g., changing weather and time of day). However, existing road driving dataset provide large-scale images acquired in either normal or adverse scenarios only, and often do not contain the road obstacles captured in the same visual domain as for the other classes. To address this, we introduce a new dataset called AVOID, the Adverse Visual Conditions Dataset for real-time obstacle detection collected in a simulated environment. AVOID consists of a large set of unexpected road obstacles located along each path captured under various weather and time conditions. Each image is coupled with the corresponding semantic and depth maps, raw and semantic LiDAR data, and waypoints, thereby supporting most visual perception tasks. We benchmark the results on high-performing real-time networks for the obstacle detection task, and also propose and conduct ablation studies using a comprehensive multi-task network for semantic segmentation, depth and waypoint prediction tasks. Finally, through experiments, our network was confirmed that exhibit the best performance.