Attention-based video masking for improving open set action recognition

Abstract

In real-world scenarios, human action recognition (HAR) is essentially an open set problem that requires a model to classify actions from known classes and detect actions from unknown classes simultaneously. However, HAR models are easily biased to static information in the video (e.g., background), which can lead to performance degradation of open set action recognition (OSAR) models. In this paper, we propose a simple framework for improving OSAR based on the video attention map extracted from the video vision transformer model. Specifically, our framework eliminates patches with static bias in video using two debiasing steps: (1) frame selection and (2) patch masking. Experimental results show that our framework achieves consistent performance improvement on multiple OSAR methods and challenging benchmarks. Furthermore, we introduce two new OSAR tasks, Kinetics-400 vs. Kinetics-600 exclusive and Kinetics-400 vs. Kinetics-700 exclusive, to validate our method in a setting close to the real-world scenario. With extensive experiments, we demonstrate the effectiveness of our attention-based masking, and in-depth analysis validates the effect of static bias on OSAR.