Real-time lyrics tracking system of singing performance with augmented reality interface

Abstract

Real-time lyric tracking aims to track the precise location of the lyrics of a given song in real-time, based on the singing input from a microphone. Current surtitle systems in concert venues are vulnerable to human error due to manual control by operators. Also, frequent eye movements are required for the audience due to current surtitle displays separated from the stage. These factors may degrade the audience’s immersion in the performance-viewing experience. In this study, we propose an automated lyric tracking model that utilizes augmented reality(AR) glasses interface for singing performances with accompaniment. Most audio-to-lyrics alignment research has focused on the separated vocals, which is not suitable for actual live performances. Previous real- time lyric tracking studies for automated surtitle operations included real-time opera tracking, but these focused more on non-musical factors, such as recitatives, noise, and applause, rather than the inherent characteristics of the singing. Additionally, these studies often assumed manual annotation work in the preprocessing stage. The proposed system utilizes a symbolic score to automate the preprocessing step and focuses on robust real-time alignment by extracting features that can capture both pitch and phonetic characteristics suitable for tracking singing performances. In particular, we show that the temporal accuracy of real- time alignment can be significantly improved when the phonetic feature extracted through the phoneme classifier is combined with chroma features. We also verified the effect and usability of the system integrated with the AR glasses interface by conducting user tests in a concert hall.