Unified spatio-temporal neural networks for contextual action understanding

Abstract

Context is important to understand human action because the same action can be interpreted in different ways, depending on context. Specifically, context works in a bottom-up or a top-down manner. On the one hand, context is formed by observing a series of actions in a bottom-up manner and affects to the recognition of the next action. On the other hand, context is predefined or planned before action execution and then used to execute an intended series of actions in a top-down manner. The type of actions that require contextual information is called contextual action. Contextual action should be comprehensively understood including bottom-up and top-down contexts. To do that, a mechanism for long-term information processing should be considered for maintaining contextual information over a long period of time. Despite the importance of contextual action understanding, the studies for contextual action understanding have not been active. Therefore, in this dissertation, we cover contextual action understanding, especially for contextual action recognition and planning, from deep neural network modeling to practical problems related to the training of deep neural networks. The contributions of this dissertation are the following.

First, we propose a unified spatio-temporal network to overcome the existing neural networks for action recognition having the difficulty in contextual action recognition. The proposed unified spatio-temporal neural network combines a spatial hierarchy and a temporal hierarchy of individual neural networks into a single neural network having a spatio-temporal hierarchy. Through the spatio-temporal hierarchy, the proposed neural network can extract low-level motion features in lower layers and high-level motion features in higher layers. Thanks to long-term processing capability, the proposed neural network shows robust recognition performance under severe dynamic occlusion, and successfully recognizes contextual action, which cannot be done by the existing neural networks.

Second, we propose a temporal normalization method to enhance the contextual processing capability and the learning speed of the unified spatio-temporal neural network. Although the unified spatio-temporal neural network has the rich spatio-temporal processing capability required for contextual action recognition, saturation functions cause a vanishing gradient problem limiting long-term processing capability and the network training is very slow because of the model complexity. Compared with existing normalization methods, the proposed temporal normalization method shows better learning acceleration and contextual processing capability. In addition, the improvement of the proposed method is further boosted by using the proposed method with existing spatial normalization methods.

Finally, we propose a unified spatio-temporal neural network based on stochastic predictive coding for planning and executing an appropriate series of actions when a specific context is given. Predictive coding framework is able to encode multimodal information, but it should predict high-dimensional sensory information, which requires huge computation. Also, in the case of the networks under deterministic predictive coding, a huge amount of training samples is required for good generalization. The proposed network reduces the computational cost by using dynamic visual attention and improves the planning performance by maintaining long-term visuospatial information on an external visuospatial memory. Furthermore, the proposed network provides good generalization with a small amount of training samples thanks to stochastic predictive coding using variational Bayes.