Real-time musical cue detection using gesture analysis for reactive accompaniment system

Abstract

In ensemble performance, visual information is not only crucial for audience evaluation of music but also essential for synchronization among performers. Especially in sections like the start of a piece or a fermata with sudden tempo changes, synchronization relies on musical cues such as a gesture and a breathing sound. However, a challenge arises in ensembles with computer-controlled pianos and human performers due to the absence of the pianist, making traditional musical cues inapplicable. This paper proposes a system that captures the performer's gesture cue in real-time through video, triggering the reactive piano accompaniment. This study is divided into two main parts. (1) First, a dataset including video, audio, and MIDI of flute and piano duets was collected to analyze the gesture cue of flutists. (2) Second, after analyzing the relationship between the flutist's gesture cue and onset timing, a system was developed that detects the gesture cue in real time and initiates the piano accompaniment at the optimized timing. User evaluation of this system was also conducted. Through this approach, the paper overcomes a limitation of previous audio-based interactive accompaniment systems by applying the gesture cue to the start of pieces and fermata passages, enabling optimized reactive accompaniment.