Deep generative classifiers for crowdsourced multiple annotator data

Abstract

Supervised classification with deep neural networks requires a large amount of labeled data. The massive cost of building such datasets has become a major bottleneck in utilizing deep learning algorithms. Recently, crowdsourcing has emerged to offer a scalable method to label massive datasets effectively. Albeit efficient, learning from such crowdsourced data is difficult since it suffers from noise in the collected labels due to varying annotator expertise. In this paper, we propose a method for learning a robust deep neural network classifier from noisy annotator data. Using a deep neural network trained with multiple annotator labels, we construct a generative classifier on top of the penultimate features of the pre-trained network. Then, we develop a robust version of the generative classifier to achieve better decision boundaries and generalization performance by jointly modeling the latent ground truth labels, deep neural network output features, and multiple annotator labels. The parameters of the robust generative classifier are estimated via Expectation-Maximization. We evaluate the proposed method on both synthetic and real multiple annotator data with complex annotator noise, where it outperforms other baselines.