Analysis of Variance: What Is Your Statistical Software Actually Doing?

Users assume statistical software packages produce accurate results. In this article, the authors systematically examined Statistical Package for the Social Sciences (SPSS) and Statistical Analysis System (SAS) for 3 analysis of variance (ANOVA) designs, mixed-effects ANOVA, fixed-effects analysis of covariance (ANCOVA), and nested ANOVA. For each model, the authors examined 3 different data sets. With the mixed-effects design, results were always correct for SPSS syntax and SAS syntax. For SPSS point-and-click, the F and p values for the random-effect were always incorrect as the wrong error term is used. With the ANCOVA design, results varied both by software package and by type of sums of squares. With the nested design, the p values for the F and multiple comparison procedure did not agree for the nonnested factor in SPSS point-and-click. Recommendations were made regarding which package to use for each design.