Integrating wearable muscle sensing for hand-based interactions in mixed reality

Abstract

This dissertation proposes a method for utilizing wearable muscle sensing technology to recognize users' hand force and touch information in the context of mixed reality (MR) environments and to enhance hand-based interactions and collaborations. Although mixed reality technologies, encompassing augmented reality (AR) and virtual reality (VR), have advanced, hand-based interactions for real-virtual 3D interactions heavily rely on visual information provided by head-mounted displays, leading to limitations that hinder natural and effective interactions. To address this, a comprehensive framework is proposed in this dissertation for integrating wearable muscle sensing technology into MR interactions and collaborations, considering the user's posture, interacting objects, and sensor signals. Furthermore, a system is developed that applies wearable electromyography (EMG) sensors to measure the hand force of a worker in an MR remote collaboration system. The impact of visualizing the worker's hand force on the collaborator's task awareness is evaluated through user studies. The visualization of hand force data positively influences task comprehension, force perception, object weight estimation, and the collaborator's sense of social presence. Moreover, this dissertation proposes a system that utilizes EMG sensors to estimate accurate touch events and touch intensity in MR interactions. The system demonstrated the capability to enable touch interactions that utilize force information, leveraging all fingers for precise and force-sensitive touch within mixed reality. In conclusion, the research findings contribute to the improvement of MR interactions and collaborations by utilizing muscle sensing technology to provide additional force information and accurate hand gesture recognition, alongside users' hand information, in mixed reality interaction and collaboration scenarios.