3D sign language handshape generation from RGB video

Abstract

This research aims to estimate 3D sign language hand shape animation from 2D RGB video with machine learning and artificial intelligence techniques. There has been great advances in 3D pose estimation research, which extracts human 3D poses from 2D RGB videos and images. However, several factors can be considered to improve when extracting 3D sign language poses. Such considerations include grammatical constraints, or eliminating noisy data. To achieve this goal, this thesis leverages 3D pose estimation technique to extract raw 3D poses from given 2D RGB video inputs. Furthermore, Triangular Moving Average based filtering is applied to eliminate noisy data. Gaussian Mixture Model based key- pose extraction component is applied to filtered data to extract significant key poses from given lexicon video. Extracted key-poses are then sorted according to intersecting point of Gaussian distributions. Lastly, extracted key-poses are compared with pre-designed sign language handshape models to configure handshape. Our system is tested on total of 1740 lexicon instances from American Sign Language users. Ablation study shows that our system was able to improve upon state-of-the-art pose estimation network.