Robust Reputation Mechanisms for Achieving Fair Compensation and Quality Assurance in Crowdcomputing

The process of distributing tasks to anonymous group of people, collecting their solutions and integrating them to produce the final result is called Crowd computing. It is a promising alternative for large and small business enterprises to accomplish many different types of tasks through geographically distributed workforce with uncontracted workforce in a la pay-per-use basis, allowing them to seemlessly scale their businesses. Two primary concerns that arise in adapting crowd sourcing to accomplish important business objectives are: (a) Ascertaining that acceptable quality of work is obtained from crowd workers (b) Assigning commensurate recognition or monetary rewards to the crowd workers to nurture and maintain a reliable workforce. In this work, we propose several mechanisms to distribute tasks to crowd workers, collect their outputs and compute final results and assign payments in a manner that addresses these two concerns simultaneously. Our methods intimately involve the use of gold tasks for which answers are pre-computed. In any set of regular tasks that are to be assigned to a crowd worker an indistinguishable set of gold tasks are intermixed. We make several contributions: (a) Our first set of schemes are based largely on evaluating the responses of the crowd workers on the gold tasks and then using them to assign appropriate weights to the crowd workers. This performance measure is used to assign appropriate weights to the crowd workers. This influence factor is then used to determine how the responses of the given crowd worker are to be assimilated with the responses of other crowd workers to compute the final answers. The compensation awarded to the crowd workers may also be made to depend on this weight. These first two schemes only use the current performance of the crowd workers to decide their compensation for the assigned set of jobs, as well as for quality assurance purpose. (b) Secondly, we propose a general reputation function which takes th- number of tasks correctly and incorrectly performed by the crowd worker, till date, and maps it to an integral value. The reputation function has several features so that it penalizes poorly performing workers, spammers and scammers in such a manner that they need to considerably perform well to be assigned a positive integral value. At the same time, no worker however much lowly performing it may have been in past, is doomed for ever by the reputation function. That is the worker can start regularly performing well and raise its reputation so that he/she is compensated well etc.. The reputation function is generic in nature that it can be employed in variety of such schemes. (c) Thirdly, we design schemes that take into account both the past performance of the worker, which is reflected by the current reputation value, and the current performance, reflected by crowd workers performance on the gold tasks, to compute the answers to the regular tasks and to decide the compensation to be awarded to the crowd workers. We also describe how response times of the crowd workers in accomplishing the tasks can be incorporated in awarding them compensations.