The effects of direct observation on student responses in the renewable energy REU program at Colorado School of Mines

The Cornell Office for Research on Evaluation (CORE) Systems Evaluation Protocol (SEP) is a step-by-step guide to evaluation, emphasizing systems modeling principles, facilitating the creation of evaluation plans with clear, specific and measurable outcomes. In 2011, CORE SEP techniques were applied to the Renewable Energy Materials Research Science and Engineering Center (REMRSEC) National Science Foundation (NSF) funded Research Experience for Undergraduate (REU) site and revealed two very different pathways leading to the critical outcome of students obtaining post-baccalaureate jobs in technical fields related to renewable energies. While the experiences within a student´s research group defined a clear evaluation pathway, a second pathway characterizing student attitudes shaped by the REU community at-large also emerged. This created an interesting question on the effects direct observation has on evaluation data. In response the REMRSEC REU SEP evaluation plan saw different implementations from 2011 to 2012. While both employed similar measures, the 2012 evaluators intervened only digitally. In this paper we summarize the REMRSEC REU and its CORE SEP evaluation plan. We conclude with comparison of the two evaluation implementations and speculate on how evaluator role affects the gathered data. Lastly, we discuss plans to hybridize previous implementations into the 2013 cycle, which we characterize in terms of evaluator role theory.