How Do Students Solve Parsons Programming Problems? -- Execution-Based vs. Line-Based Feedback

In large introductory programming classes, there typically are no resources for adequate individual guidance. Automatic feedback for programming tasks can facilitate students´ learning by allowing them to get immediate individual feedback regardless of time and place. This paper presents a study on how the type of automatic feedback in Parsons problems affects how students solve them. Students on their first programming class were divided into two groups and, in two assignments, each group in turn received different type of feedback. The type of feedback had an effect on how students constructed their programs and how quickly they were able to complete them. With feedback based on execution as opposed to the visible arrangement of code, the programs were more frequently executable when feedback was requested and, overall, feedback was requested less frequently. Based on the analysis, we discuss possible future improvements to automatic feedback in this type of an assignment.