Mapping multiple modalities to empower novice designers

Abstract

Software applications have the potential to expand the design capabilities of novices. Adequately leveraging these software applications, however, can be prohibitive for novices as the input that software is designed to accept and understand is restricted. Novices must be able to adapt their intended input according to the software's constraints: condense it into the limited input modalities supported by the software and translate it to the software's internal language. The burden and learning curve of this process introduces an additional barrier to entry for novice designers. Consequently, novices rarely reach the potential that software applications grant. This thesis proposes a high-level approach to lower this barrier: support multimodal input natural to novices, but computationally map the multiple modalities to enable design tasks through the software. Through this approach, novices can leverage multiple modalities to flexibly express themselves without considering how to map their input to the software's accepted input. By extracting and combining the rich information embedded in it, the multimodal data can be transformed into interaction widgets that afford the software's design tasks. In this thesis, I present techniques for applying this approach with multimodal input of voice and clicks, and mapping operations according to temporal and contextual dimensions. Then, I present two interfaces that employ these techniques to facilitate two design-centric tasks involving novices: asynchronous UI design by collaborative student teams, and website styling by end-users.