Real-time pitch tracking using weakly supervised convolutional recurrent neural network

Abstract

Existing audio analysis algorithms are focused on using by composers. Performers also use similar techniques, but algorithms should work in real-time without post processing. Pitch tracking is one of the famous technology that is applied to various audio signal processing technologies. Especially for synthesizing new sound real-time high performance pitch tracking is necessary for performers. Digital signal processing(DSP) based pitch tracking algorithms like YIN or probabilistic YIN algorithm shows high accuracy and they are generally used for pitch tracking tasks. Still those algorithms are difficult to cope with various recording environments and have a long analysis time. In this paper, we propose a pitch analysis algorithm using neural network which can learn various recording environments based on data and reduce the number of operations. Especially we adopt convolutional neural network and convolutional recurrent neural network which show high pitch tracking accuracy. Also we applied post processing based on average mean difference function which is used in DSP pitch tracking and help finding fine pitch. The problem of neural network is that it needs large enough data to be trained. Here we propose weakly supervised learning idea which obtain annotation from DSP algorithm especially using PYIN algorithm. We found that the prediction from DSP annotation shows close accuracy compared to the prediction from human annotation. Though these process user can obtain continuous pitches complete automatically. We made our own dataset to train pitch tracking. The dataset is consist of jazz and blues style guitar solo. We compared result among several different setup networks and also compared our method with DSP algorithms in accuracy and computation speed. We mainly focused on voiced samples. Experiment is done using same test set from dataset and computing environment.