Effect of being observed on the reliability of responses in crowdsourcing micro-task platforms

In this paper, the reliability of responses collected in two crowdsourcing studies is compared. Two methods to evaluate the reliability of responses (one noticeable and one unnoticeable method for workers) have been employed. The unnoticeable method is included in both studies; the noticeable method is employed in one study only. The study containing the noticeable reliability check method resulted in a higher consistency of the responses than the other one. We assume that the difference is a result of the obvious reliability check method: Workers improve their performance due to the awareness that they are being observed.