SWAG DEMO: SMART WATCH ASSISTED GESTURE INTERACTION FOR MIXED REALITY HEAD-MOUNTED DISPLAYS

Abstract

In this demonstration, we will show a prototype system with sensor fusion approach to robustly track 6 degrees of freedom of hand movement and support intuitive hand gesture interaction and 3D object manipulation for Mixed Reality head -mounted displays. Robust tracking of hand and finger with egocentric camera remains a challenging problem, especially with self -occlusion for example, when user tries to grab a virtual object in midair by closing the palm Our approach leverages the use of a common smart watch worn on the wrist to provide a more reliable palm and wrist orientation data, while fusing the data with camera to achieve robust hand motion and orientation far interaction.