Audio-visual learning with semantically similar samples

Abstract

Instance discrimination-based contrastive learning is the learning method that contrasts the positive and negative pair. It assumes that the negative pair should contain different semantic information. However, the assumption only holds because of the random construction of the training batch. Intuitively, this faulty negative pair disturb the training and degrade the model performance. This work aims to solve the faulty negative problem for in- stance discrimination-based Audio-Visual Learning. Existing audio-visual works employ contrastive learning by assigning corresponding audio-visual pairs from the same source as positive while randomly mismatched pairs as negatives. As aforementioned general instance discrimination-based contrastive learning, these negative pairs may contain semantically matched audio-visual information. The key contribution of this work is showing that semantically similar samples can compensate for the effect of faulty negative pairs. Our approach incorporates semantically similar samples into a contrastive learning objective directly. It is applied to two tasks: Audio-Visual Sound Source Localization and Visually Grounded Speech. We demonstrate the effectiveness of our approach to the tasks.