CompositeGesture: Creating Custom Gesture Interfaces with Multiple Mobile or Wearable Devices

Abstract

A growing need exists for gesture customization in mobile interaction design. However, two problems hinder gesture customization: (1) lack of sensors available in daily mobile life and (2) difficulty with defining three-dimensional gestures intuitively. In order to resolve these problems, we present CompositeGesture, a mobile application that allows users to customize gestures using the sensors of multiple mobile or wearable devices. Utilizing motion sensors in mobile devices, CompositeGesture allows a user to create personalized gestures. It provides an intuitive gesture creation scheme based on visual coherence of motion representation, definition, and recognition. Applying a glass box approach in gesture definition, users can clearly and intuitively understand how the system works. It allows a user to easily specify a function of mobile application to be executed with a defined gesture. An exploratory user study raised three issues in gesture for command, engaged experience, and social expression. This research proposes a novel way of customizing gestures and prototyping interactive products with complex, personalized gesture interfaces.