Designing argumentation in peer review: Patterns of evaluation in the AltCHI open peer review forum

Author

Rughinis, Razvan

Author_Institution

Fac. of Autom. Control & Comput., Univ. Politeh. of Bucharest, Bucharest, Romania

fYear

2013

fDate

19-22 June 2013

Firstpage

1

Lastpage

6

Abstract

We examine patterns of evaluation on the open peer review forum of the AltCHI panel of the ACM SIGCHI Conference on Human Factors in Computing Systems, 2012-2013. By analyzing a dataset including all available reviews posted on the AltCHI online platforms, we find that author reviewers are more critical than external commentators, and later submission reviews are more favorable than earlier ones. We find substantial inter-reviewer agreement, partly explainable through persuasive convergence and anchoring effects. Final article acceptance through juried selection is strongly correlated with the average grade in open review, while the number of article reviews is less consequential for selection. The communication design of AltCHI open review balances several conflicting participation incentives, in order to support juried selection for atypical and potentially controversial papers.

Keywords

Internet; electronic publishing; reviews; ACM SIGCHI conference; AltCHI online platforms; AltCHI open peer review forum; AltCHI open review balance; AltCHI panel; anchoring effects; article reviews; communication design; computing systems; human factors; juried selection; participation incentives; persuasive convergence; substantial inter-reviewer agreement; Convergence; Correlation; Correlation coefficient; Human factors; Portals; Sociology; AltCHI; argumentation; design; open peer-review;

fLanguage

English

Publisher

ieee

Conference_Titel

Information Systems and Technologies (CISTI), 2013 8th Iberian Conference on

Conference_Location

Lisboa

Type

conf

Filename

6615778