Retrieval for footsteps sound assets and auto-triggering of humanoid character footsteps in in-game cutscenes

Abstract

A need to develop a footsteps sound assets retrieval and auto-triggering method for in-game cutscene Foley work was identified. Footsteps assets retrieval and auto-triggering improve the efficiency of Foley artist's work. This paper introduces how to search for footsteps sound assets through metric learning and how to predict footsteps sound on a frame-by-frame. Footsteps sound asset retrieval model trained with metric learning by structure preserving. Retrieval performance has been improved using video and audio features extracted from the pre-trained model learned through audio-visual correspondence as training data. Footsteps sound auto-triggering model was trained with skeleton data and footsteps sound origin location data labeled via beat predominant local pulse. The proposed method allows us to search for appropriate footsteps sound assets for silent game videos, and specify where to insert them.