A classifier-based association test for imbalanced data derived from prediction theory

How can we test for group differences in multidimensional input patterns, such as functional magnetic resonance imaging measurements or gene expression values? One solution is to split the available data into training and test set, and to estimate the generalization accuracy of a classifier that predicts the group variable from the input pattern. If this lies significantly above chance level, we can reject the null hypothesis of no association. This test is straightforward for balanced data, where all groups are equally frequent in the data set. However, data sets collected in observational studies are often imbalanced. Then accuracy is no longer a suitable measure of performance, and balanced accuracy should be used instead. In this paper, we give an overview on existing analytical tests and use the framework of prediction theory to derive a new test for the balanced accuracy of a classifier. We then use numerical simulations to evaluate the type I error rate and the power of two tests for imbalanced data.