Pluggable reputation systems for peer review: A web-service approach

Peer review has long been used in education to provide students more timely feedback and allow them to learn from each other´s work. In large courses and MOOCs, there is also interest in having students determine, or help determine, their classmates´ grades. This requires a way to tell which peer reviewers´ scores are credible. This can be done by comparing scores assigned by different reviewers with each other, and with scores that the instructor would have assigned. For this reason, several reputation systems have been designed; but until now, they have not been compared with each other, so we have no information about which performs best. To make the reputation algorithms pluggable for different peer-review system, we are carrying out a project to develop a reputation web service. This paper compares two reputation algorithms, each of which has two versions, and reports on our efforts to make them “pluggable,” so they can easily be adopted by different peer-review systems. Toward this end, we have defined a Peer-Review Markup Language (PRML), which is a generic schema for data sharing among different peer-review systems.