Work in progress - improving interrater agreement used to measure learning outcomes

The Inter-professional Project Program (IPRO) at IIT hosts a one-day evaluation event (IPRO Day) every semester, during which 70-120 judges (~50% new each semester) judge 35-45 teams averaging 12 students each. The judges, in groups of three to five, rate ten criteria per team on a 1-5 scale for either group oral presentations or exhibits. While the statistics of measuring interrater agreement are well documented it is not completely clear from previous research how best to improve such agreement except for assertions that rater training is beneficial. Over the past two years, we have instituted several interventions to improve interrater agreement on IPRO Day: using experienced judges to coordinate each judging group, rewriting judging forms to match the learning objectives, developing rubrics to guide judging, and conducting orientation sessions with the judges. This study evaluates the effectiveness of these interventions by comparing judging group agreement results from semesters prior to the change with the semesters following the change. These results having broad applicability to all judging review panels.