Temporal Convolutional Networks for Offline and Online Action Detection

Abstract

Detecting actions in untrimmed videos is an important yet challenging task. In this dissertation, we propose two new approaches for offline and online action detection. First, we introduce a temporal convolutional network, called Temporal Relation Network (TRN), for offline action detection. Specifically, TRN produces a 2D relatedness score map based on a novel concept of snippet relatedness representing which snippets are related to a specific action instances, and then TRN evaluates the action confidence scores of the temporal intervals and refines their boundaries to obtain temporal action proposals. Our relatedness score map enables the generation of various temporal intervals reliably covering most action instances with high overlap. On two benchmark datasets, THUMOS-14 and ActivityNet-1.3, the proposed method outperforms state-of-the-art methods for temporal action proposal generation. Furthermore, TRN leads to significant improvements in temporal action detection by combining existing action classification networks.Second, we present a novel method, named Temporal Filtering Network (TFN), for online action detection. TFN aims to distinguish between relevant and irrelevant information from a streaming, untrimmed video. To this end, we introduce a filtering module to learn relevance scores indicating how relevant the information is to a current action. Our filtering module emphasizes the relevant information to a current action, while it filters out the information of background and unrelated actions. We conduct extensive experiments on THUMOS-14 and TVSeries datasets, where the proposed method outperforms state-of-the-art methods by a large margin. We also show the effectiveness of the filtering module through comprehensive ablation studies.