Scalable Expert Selection When Learning from Noisy Labelers

In a supervised learning context, various methods have been proposed to learning from different labelers. Very recently, the problem has shifted towards ranking and filtering low-quality annotators, and estimating the consensus labels based only on the remaining experts, i.e, annotators that provide high quality annotations. In this paper, we propose a novel approach to address this issue. Our solution is based on a probabilistic method where a combination of two metrics, a probabilistic score and an entropy measure, are integrated in order to iteratively select the experts and estimate the labels based only on the selected annotators.