Robust feature alignment for mitigating negative transfer in domain adaptation

Abstract

Domain Adaptation (DA) involves leveraging knowledge from a labeled source domain to an unlabeled target domain. This adaptation implicitly assumes that the source domain and the target domain are from different distributions, making feature alignment a crucial aspect of the domain adaptation problem. Aligning the feature distributions allows the model to perform well on the unlabeled target domain by incorporating knowledge from the source domain. However, in realistic scenarios, certain factors can degrade the performance of domain adaptation models, leading to negative transfer, where the adapted model performs worse than the base model without adaptation. Therefore, this thesis investigates the factors contributing to negative transfer in domain adaptation under realistic scenarios and explores their impact on the performance of the target domain. To address negative transfer, we propose methods for robust feature alignment, demonstrating their effectiveness in improving performance. Specifically, we address negative transfer in two realistic scenarios of domain adaptation where existing feature alignment methods face challenges. Firstly, we examine Open-Set Domain Adaptation (OSDA), addressing class-set mismatches where the target domain introduces unknown classes absent in the source domain. Feature alignment in OSDA can degrade performance due to negative transfer, aligning unwanted features from unknown classes with the source domain. Therefore, we identify the misalignment of unknown instances as a factor for negative transfer. To mitigate this, we propose unknown-aware domain adversarial learning for OSDA, segregating the target-unknown distribution during feature alignment. Theoretical analysis demonstrates the importance of explicitly segregating unknown features in OSDA. Secondly, we explore negative transfer in Test-Time Adaptation (TTA), where adaptation must occur in an online manner without access to the source dataset. Existing TTA approaches face conflicts among individual objectives during online updates, leading to negative transfer. We identify conflicting gradients as a factor for negative transfer in TTA and propose an approach to update model parameters towards Pareto-Optimality across all individual objectives in the current batch. We introduce an extended Pareto optimization incorporating sharpness-aware minimization to anticipate unexpected distribution shifts during testing time, demonstrating its effectiveness through empirical analysis and experiments. In summary, this thesis focuses on OSDA and TTA scenarios, identifying factors contributing to negative transfer and proposing robust feature alignment approaches to mitigate these challenges. Through various experiments, we establish that robust feature alignments lead to enhanced performance in domain adaptation by mitigating negative transfer.