Understanding grasp intention with gaze and hand gesture input for adaptive virtual object interaction

Abstract

This study explores how gaze and hand gesture inputs can be used for intention-aware adaptive virtual grasping interactions. A conventional pinch-grab method allows virtual objects to be grabbed easily with a pinch on any part of the object, making it suitable for simple moving tasks. However, it falls short of making a precise pose refinement which is an essential process to grab by the handle to use with its original purpose as in the real world. To recognize grasp intentions and enable appropriate grasping techniques, we investigated the feasibility of using gaze and hand gesture inputs as cues indicating the intention to move or use virtual objects. Our findings revealed that while implicit gaze behavior alone has limitations in determining grasp intentions, incorporating hand movements can enable better intention differentiation and intuitive interactions. Based on these findings, we implemented and evaluated adaptive virtual grasping techniques each triggered by explicit gaze-based selection and hand gestures. Through this research, we propose design implications for gaze and hand gesture-based grasp intention-aware adaptive interactions. We anticipate that this approach will elevate virtual objects beyond simple digital content to become versatile functional tools that can be utilized freely and diversely, similar to their real-world counterparts.