The impact of background knowledge on test performance: A multivariate G-theory approach

Validity has been declared to be the single most important consideration in language testing and educational measurement. Messick (1989) famously identified construct-irrelevant variance as one of the major threats against test validity. Hence, test users ought to make every possible effort to make sure that test scores are not unduly affected by construct-irrelevant factors. This study applied multivariate Generalizability Theory to examine the effect of academic background on test scores obtained from the Iranian national university entrance exam. The results revealed that the relative contribution of the various sources of variance was not the same across the academic background groups. In addition, dependability indices were significantly different across the groupings. Furthermore, the decision studies revealed that the groups do not necessarily need to take the same number of items so that a high reliability is obtained. Overall, the results indicate that academic background exerts a remarkable influence on the dependability of the scores.