Temporal procrustes alignment framework for 3D human pose and shape estimation from video

Abstract

This dissertation considers the task of model-based 3D human pose and shape estimation from 2D monocular RGB video. Human pose and shape estimation takes a monocular image containing a person as input and returns that person’s joint position and body shapes. To obtain model-based 3D results, depth estimation is added to the 2D joint location estimation, which is used with the body shapes to generate 3D human mesh model parameters. For the case where the input is a video, the above process is done for each image frame of a video. Although the estimated pose and shape of a person may seem accurate for each frame of image, simply listing the estimated results of the images does not give smooth results for they show jitter along the predictions due to the uncertainty of the joint positions. To solve this problem, previous methods have made progress by improving networks to consider the temporal consistency of human motions in sequential frames by supervising the average acceleration of joints for each frame. After maintaining temporal consistency, however, geometric misalignments within the sequence of joints are observed. Geometric misalignment refers to the steady deviation between the geometric path drawn by a sequence of predicted joints and that of ground-truth joints. To this end, we propose Temporal Procrustes Alignment (TPA) framework, which is a model-agnostic framework that mitigates geometric misalignments by performing group-wise sequential learning of every joint’s movement paths. While previous methods rely entirely on per-frame supervision for accuracy, our framework can supervise sequential accuracy by performing Procrustes Analysis to the sequence of predicted joints. Experiments show that TPA framework mitigates the misalignment of the results without damaging their temporal consistency, advancing the network to mostly exceed the previous state-of-the-art performances on benchmark datasets in both per-frame accuracy and video smoothness metric.