Evaluating the contribution of different item features to the effect size of the gender difference in three-dimensional mental rotation using automatic item generation

In complex three-dimensional mental rotation tasks males have been reported to score up to one standard deviation higher than females. However, this effect size estimate could be compromised by the presence of gender bias at the item level, which calls the validity of purely quantitative performance comparisons into question. We hypothesized that the effect of gender bias at the level of distinct item design features could lead to either an over- or underestimation of reported effect sizes of the gender difference in three-dimensional mental rotation. Using automatic item generation we conducted a series of psychometric experiments in which we independently manipulated one out of four different item design features that have exhibited a gender bias in the previous studies (study 1). This was done in a between-subjects design. The results indicated that gender bias caused by item design features linked to the perceptual stadium of mental rotation led to an overestimation of the effect size of the gender difference while item design features associated with the encoding and transformational stadium resulted in an underestimation of the effect size of the gender difference. In study 2 we tested the hypothesis that the gender difference still remains while controlling for the item design features causing gender bias. The results suggest that a significant portion of the gender difference may be attributable to perceptual and encoding processes involved in mental rotation.