AILA: Attentive Interactive Labeling Assistant for Document Classification through Attention-based Deep Neural Networks

Abstract

Document labeling is a critical step in building various machine learning applications. However, the step can be time-consuming and arduous, requiring a significant amount of human effort. To support an efficient document labeling environment, we present a system called Attentive Interactive Labeling Assistant (AILA). At its core, AILA uses Interactive Attention Module (IAM), a novel module that visually highlights words in a document that labelers may pay attention to when labeling a document. IAM utilizes attention-based Deep Neural Networks, which not only support a prediction of which words to highlight, but also enable labelers to indicate words that should be assigned high attention weights while labeling to improve the future quality of word prediction. We evaluated the labeling efficiency and accuracy by comparing the conditions with and without IAM in our study. The results showed that the participants’ labeling efficiency increased significantly under the condition with IAM than under the condition without IAM, while the two conditions maintained roughly the same labeling accuracy.