A Weighted Aggregation Rule in Crowdsourcing Systems for High Result Accuracy

Many challenging problems could be better solved by exploiting crowdsourcing platforms than traditional machine-based methods. However, data quality in crowdsourcing applications has become a crucial aspect since crowdsourcing workers may have different capabilities. In this paper, we propose a novel weighted aggregation rule (WAR) to improve the result accuracy in crowdsourcing systems. According to the agreement of answers given by the workers, we classify all the tasks into the high-agreement tasks and low-agreement tasks. For the high-agreement tasks, we use simple majority voting to select the correct answer while ensuring the result accuracy. For the low-agreement tasks, we adopt weighted majority voting strategy, which assigns a weight for each worker according to his performance on the high-agreement tasks. We evaluate the effectiveness of our proposed method using three real-world datasets on AMT. The experimental results show that our method achieves excellent result accuracy.