Learning with Guaranteed Label Quality

In supervised learning, label quality is crucial for learning performance. However, noise is ubiquitous in labels provided by oracles in active learning. To rule out its negative influence, multiple-oracles have been proposed. However, unrealistic assumptions (such as the evenly distributed noise level of oracles) have been made to restrict the learning algorithms for real-world applications. In this paper, we propose a learning algorithm, c-certainty, to guarantee the label quality, and allow the noise level of oracles to be example-dependent. Furthermore, we develop an effective learning algorithm which is able to select the more accurate oracles to query. The experiment results show that the learning strategy developed in this paper outperforms other learning algorithms significantly.