Audio-text word representations for open-vocabulary keyword spotting

Abstract

Open-vocabulary keyword spotting (KWS) is a technology detecting an occurrence of arbitrary keywords from input audio, and it has high research value in that users can customize their keywords while liberated from a pre-defined set of keywords of conventional KWS. However, compared to conventional KWS widely used in mobile devices, open-vocabulary KWS needs much performance improvement before it can be practically applicable. In this dissertation, we analyze the optimistic development of recent deep learning approaches utilizing audio and text jointly to represent words through the roles of two modalities when mapped to the same embedding space, and it is confirmed with a proposed Decoder-Sharing method. We extend the framework of audio-text representation into proxy-based deep metric learning (DML) and propose an Asymmetric-Proxy loss by exploring the optimal combination of existing DML loss functions. In addition, we introduce an Adaptive Margin and Scale method where class-wise learnable parameters dynamically change according to the training progress, which shows significant improvement in generalization performance. Finally, we propose a Monotonic-Aligned Audio-Text loss to resolve the data segmentation problem that embedding-based open-vocabulary KWS approaches suffer at inference.