Cross-modal knowledge distillation for one-shot human action recognition

Abstract

Human action recognition (HAR) aims to understand human behaviors and predict the correct answer to each action using various visual information such as RGB video, infrared video, depth information video, or skeleton information as input data. In action recognition, the action may be expressed by different movements depending on the performers or interpreted as different actions in a specific domain. Such expressions make it challenging to prepare sufficient data for the learning of action recognition models. Thus, we consider an efficient method that can be trained with few samples and applied its potential features to other domains such as knowledge distillation. In this paper, we propose a teacher-student network to learn the representations from the given actions based on the skeleton sequences and textual information describing each action. Our teacher network consists of two encoders: a skeleton encoder, which is a graph-based model to fit the structure of skeletons, and a text encoder which is pre-trained with large-scale datasets. The teacher network uses the skeleton sequences and additional textual information of the synonyms of the action labels to provide cross-modality to the student network. Furthermore, the student network contains only a skeleton encoder same as the teacher to learn the semantic relationships guided by the knowledge of the teacher. Experiments on one-shot HAR using the public dataset NTU RGB+D120 demonstrate the state-of-the-art performance of the proposed method.