Unsupervised test-time adaptation to overcome domain shift in the wild

Abstract

Despite recent advances in deep learning, deep neural networks often suffer from performance degradation when the train and test domains differ significantly. Among several tasks addressing such domain shifts, Test-time adaptation (TTA) has attracted significant attention due to its practical properties, which enable the adaptation of a pre-trained model to a new domain with only unlabeled target dataset during the inference stage.

Prior works on TTA assume that the target dataset comes from the same distribution, and model adaptation is performed for a short period. However, TTA may be carried out under two conditions that may arise in reality: (1) the target domain contains multiple subdomains (\ie, compound domain (CD)) that are sufficiently distinctive from each other, or (2) long-term adaptation to continually changing domains is performed in edge devices that are memory constrained. In this thesis, we present methods and analysis to solve the problems that may occur in each of the above two conditions.

To tackle the problem in the first condition, we assume that the compound domain might occur cyclically, such as repeated driving environments over time (\eg, daytime, twilight, and night). One naive approach is to perform domain-specific adaptation, where several models are constructed, and only test images from similar domains are used for each model adaptation. However, such a domain-specific TTA may not be feasible under the scenario that the model can not access domain labels. Hence, we propose an online clustering algorithm that allow the model to obtain pseudo-domain label and accumulate knowledge of cyclic domains. Moreover, we attempt to boost its performance with adaptation loss denoising by considering the test image-wise similarity with the source distribution. To do so, we achieve reliable adaptation performance under compound and cyclical domains.

For the second condition, we present an effective and efficient approach that improves adaptation performance in continually changing target domains. We introduce novel back-to-the source regularization that helps our newly proposed meta networks regularized by the protected source knowledge distilled from frozen original networks. With negligible computational overhead, this regularization prevents error accumulation and catastrophic forgetting, resulting in stable performance even in long-term test-time adaptation. Moreover, to help TTA to be implemented in edge devices, we further present adaptive meta networks which allow original networks not to store large sizes of activation required for backpropagation, which enables our approach to perform memory efficiently. We demonstrate that our strategy outperforms other state-of-the-art methods on various benchmarks for image classification and semantic segmentation tasks.