Gesture-based interaction prototyping for designers

Abstract

Gesture-based interaction in interaction design has great potential due to its richness and naturalness. Prototyping of gesture-based interaction is important for exploring and developing ideas. However, prototyping of gesture-based interaction has been expensive and difficult for designers due to its technical complexity and a lack of suitable systems. The goal of this dissertation is to suggest and evaluate a series of gesture prototyping systems that allow designers to easily and quickly design, explore, and test gesture-based interactions.

Literature about the design process, design prototyping, gesture interface, and existing gesture interaction prototyping platforms was reviewed. I order to support prototyping by designers, interaction prototyping systems should provide both a high ceiling and a low threshold. Existing gesture authoring toolkits were not suitable for exploring various design ideas.

The general design directions for suggested systems were to utilize designers’ competence in visual language, to utilize familiarity with CAD systems, to support quick and easy design modification, and to provide versatile and detailed programming. A framework of design spaces of gesture-based interaction prototyping systems was proposed, and three systems were suggested based on the framework. The first system was developed to propose and evaluate the experimental concept of metaphor-based geometric gesture authoring and interaction itemization. The second and third systems were developed to propose prototyping systems solving realistic and practical design problems―cross-device gestures and IoT interactions.

EventHurdle was designed to allow hurdle-metaphor-based gesture authoring and interaction prototyping with itemized gesture events. Experiment participants highly valued that the system allowed iterative design explorations. M.Gesture was designed to define device gestures using multiple mobile devices. It used the metaphor of mass-spring to visualize the behavior of an accelerometer. Evaluation revealed that the system was easy to use and learn. Most participants could compose given gestures. The hybrid algorithm marked better recognition performances than conventional dynamic time warping. M.Connect provided a gesture-based IoT interaction prototyping environment for designers in conjunction with M.Gesture. High-level trigger-action rule creation allowed easy exploration of diverse ideas. Low-level programming raised the ceiling. Gesture attributes of input gestures were visually mapped on a space, and users could filter them by creating masks. The user study found that the system was easy to understand and use by previous users of Processing. It allowed a quick exploration of trigger-action combinations, while low-level coding provided specification of detail behaviors.

The suggested systems successfully supported gesture-based interaction prototyping. The hurdle-based visual gesture authoring system was proposed and facilitated intuitive gesture design and modification. The mass-spring visualization technique allowed users to visually understand the sensor’s behavior. Gesture attribute monitoring and filtering let users specify detail behaviors in response to gesture attributes. The visual trigger-action rule allowed users to explore diverse interaction ideas. The systems presented contribute to interaction design by allowing designer-friendly gesture design and interaction prototyping. Their designs and approaches can be applied to other types of interaction design and human-computer interaction.