Quantifying learning from web-based course materials using different pre and post tests

Engineering instructors seek to gauge the effectiveness of their instruction. One gauge has been to use standardized tests, such as concept inventories and to quantify learning as the change in score over the semester. Here we question whether that approach is always the best practice for gauging the effect of instruction, and we propose an alternative of administering different tests at the start and end of the semester. In particular, to gauge the influence of one aspect of instruction, the use of interactive web-based course materials that had been developed for Statics, we administered the Force Concept Inventory at the start of the course, and the Statics Concept Inventory at the end of the course. Correlations and then linear regression were applied to study how conceptual knowledge measured at the end of the course depended on conceptual knowledge measured at the start and the amount of use of the web-based courseware. Usage of the web-based courseware was found to promote conceptual knowledge at the end of the course in a statistically significant way only after accounting for initial knowledge as judged by the different conceptual test administered at the start of the course. Thus, it is not necessary to measure gain on one test; instead each test should capture well the variation in relevant ability across students at the time the test is administered.