Evaluation of Statistical Inference Tests Applied to Subjective Audio Quality Data With Small Sample Size

Monte-Carlo simulations of statistical inference tests were performed to assess type 1 (false rejection) and type 2 (false non-rejection) error rates associated with subjective audio quality data as a function of sample size. Samples were generated by randomly drawing data from large-scale subjective audio quality tests. Null hypotheses were simulated by equalizing population means followed by pooling. The Null hypothesis rejection rates were determined for a parametric t test, as well as a non-parametric (permutation) test and compared to rejection rates based on analytical expressions and empirical distributions of the sample means and medians. The results indicated that pairwise comparisons are beneficial for high power and to obtain type I error rates that are close to the nominal value of 5%. The pairwise inferences can be realized by a parametric, pairwise t test or by a non-parametric permutation test, provided that for the latter, only pairwise permutations are executed. Although the observations from this study cannot be generalized for arbitrary data sets, the results do indicate that a pairwise, non-parametric resampling test is an interesting candidate for the statistical analysis of subjective quality data due to the absence of any requirements on data distributions and its relatively accurate performance in terms of Null hypothesis rejection rates.