End-to-end supervised semantic segmentation with contrastive learning for self-driving under adverse weather

Abstract

Scene understanding tasks such as semantic segmentation and disparity prediction have recently become crucial for self-driving vehicles. In particular, real-time semantic segmentation is indispensable for intelligent self-driving agents in recognizing roadside objects in the driving area. As previous studies have proposed various model architectures that are heavy-weight and non-real-time inference speed, and require far significant hardware resources for training and practical deployment, we present a novel contrastive approach to improve the semantic segmentation performance of a light-weight model for autonomous driving, particularly under adverse weather conditions such as fog, nighttime, rain, and snow. The proposed contrastive learning approach exploits both image-level and pixel-level labels in an end-to-end supervised learning pipeline, where image-level labels are the specific weather conditions and can be easily inferred from the given RGB images. We validate the effectiveness of our contrastive approach through various experiments on the ACDC dataset containing images of 1920×1080 resolution, and find that it achieves up to 66.7 FPS at inference with 2048×1024 Cityscapes benchmark dataset resolution, using RTX 3080 Mobile GPU. In addition, we conduct ablation studies to demonstrate how each component of our method facilitates learning of semantic segmentation maps.