Transcription and modeling of expressive piano performance using machine learning

Abstract

In this thesis, we propose machine-learning-based systems for transcriptions and computational modeling of piano performance. The performance transcription is defined as a quantization of how a pianist performed given music score, in terms of tempo and dynamics. The Dynamics can be transcribed by non-negative matrix factorization, which decomposes audio spectrogram into spectral templates of pitch and its activation over time. We employ score information as a constraint for the update of NMF to improve the performance of the algorithm. Also, we propose a method to overcome the limitation that note intensities are greatly affected by recording gain. We trained a deep neural network to estimate the key velocity of each note from the note-separated spectrogram derived from NMF. The gain of the note-separated spectrogram is normalized so that the neural network can estimate the key velocity by focusing on the timbral aspect rather than the intensity. For the other topic, we propose a performance modeling system using a deep neural network by using quantized performance. A performance modeling system is a system for creating human-like expressive performances for a given score input. To train the modeling system, we introduce a data set consisting of score and performance, and we propose a score and performance feature scheme. We also propose a system that uses graph neural network by presenting a graphical representation of musical score rather than a 1-d sequence or a 2-d matrix. Lastly, we present the example analysis of distinguished pianists from their audio recordings using our transcription and modeling system.