Budget distribution for efficient data labeling over crowdsourcing system

Abstract

We consider query-based data labeling problem in which, the goal is to classify k objects in database into binary attributes. Queries are designed using the following rule. First, randomly select query difficulty(d) number of objects. Next, ask whether those objects have an even or odd number for the given attribute. Designed queries are distributed to workers using a crowdsourcing system. We consider two system models in this paper. First is crowdsourcing erasure model. In the erasure model, workers either provides the correct answer for a query if he/she knows the answer or, refuses to answer if he/she is unsure about the answer. The second is a crowdsourcing error model. In the error model, a worker always supplies an answer, but the answer can be right or wrong. In this paper, we consider the case of multiple worker groups. Workers in the same group have the same performance on queries and the same cost for raising a query. However, workers in different groups show a different performance on queries and a different cost for raising a query. In this situation, depending on how we allocate queries to each group, the total cost used to label objects may vary. In this paper, our goal is to find the optimal distribution of queries for each group to minimize the total cost of classifying object attributes.