Hand posture and electromyography based multimodal hand pressure estimation for hand-object interaction

Abstract

This research presents a novel hand pressure estimation framework that utilizes multimodal inputs, including vision-aided 3D hand pose and wearable-based muscle activation. It supports whole-hand pressure estimation for various hand-object interaction contexts, overcoming fixed-hand pose limitations. Proposed approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG to offer a continuous and comprehensive understanding of hand dynamics to estimate pressures from fingertips to the palm. In this study, a multimodal data collection system has been developed that combines a pressure glove, an sEMG armband, and a markerless finger-tracking module. A comprehensive dataset was created from 20 participants, capturing synchronized data of hand posture, sEMG signals, and exerted hand pressure across various hand postures and hand-object interaction scenarios using the collection system. Proposed framework enables precise hand pressure estimation in complex and natural interaction scenarios and significantly mitigates the limitations of traditional sEMG-based or vision-based methods by integrating 3D hand posture information with sEMG signals.