Deepening the understanding of designer’s behavior and challenges in prompt-based prototyping with a proposition for a supportive agent

Abstract

Large Language Models (LLMs) enable non-AI expert designers to rapidly prototype AI functions through natural language. However, there is a limited understanding of how designers incorporate their design knowledge and considerations of potential end-users while prototyping AI features through prompt engineering. We conducted a design workshop where participants envisioned and implemented novel AI functionalities using GPT-3. Our findings indicate that designers consider how end-users might interact with the AI features in a real-world context. From these insights, this paper suggests the potential of a supportive agent that automatically analyzes prompts, identifies potential issues, and suggests revisions. To demonstrate the value and feasibility of the agent, we present five key use cases that help designers, including clarifying ambiguous expressions, testing their prototypes with diverse input values, and simulating the experience of potential end-users. Notably, within these use cases, we have developed an LLM-based agent that serves two primary purposes: facilitating prompt evaluation by testing it with diverse input values and simulating the user experience for potential end-users. We conducted a preliminary study to assess the quality of information provided by the agent and explore improvements in the interaction. Our findings indicate that the agent effectively reduces the cognitive load during prompt testing and user experience envisioning. However, there is still room for enhancing the agent's controllability and the relevance and coverage of its suggested information. In light of these findings, this paper also delves into the role of LLMs in the context of AI prototyping.