The use of balance-aware subsampling for bioinformatics datasets

A major challenge facing data-mining practitioners in the field of bioinformatics is class imbalance, which occurs when instances of one class (called the majority class) vastly outnumber instances of the other (minority) classes. This can result in models with increased bias towards the majority class (minority-class instances predicted as being in the majority class). Data sampling, a process which changes the dataset through removing or adding instances to improve the class balance, can be used to improve the performance of such models on imbalanced data. However, it is not clear what target balance level should be used with data sampling, and what influence class imbalance alone has on classification performance (compared to other issues such as difficulty of learning from the data and dataset size). To resolve this, we propose the Balance-Aware Subsampling technique, which allows researchers to directly compare different balance levels of a dataset while keeping all other factors (such as dataset size and the actual dataset in question) constant. Thus, any changes in performance can be attributed solely to the chosen balance level. We demonstrate this technique using six datasets from the field of bioinformatics, and we also consider three different subsample sizes (that is, the size of the dataset used for building a model) so we can observe the effect of this parameter on classification performance. Our results show that within each level of class imbalance, the average AUC value increases as the subsample size increases. The key exception is the 20:80 (minority:majority) balance level, for which the average AUC value decreases as the subsample size increases from 80 to 120. We also find that within each subsample size, the average AUC value increases as the minority distribution increases, although this does not completely hold for subsample size 40 (in which case, the Näıve Bayes and Random Forest learners show greater performance at the 3- :65 balance level than at 50:50), and in general there is not a significant improvement between the 35:65 and 50:50 balance levels. Overall, by using Balance-Aware Subsampling, we are able to directly observe how class imbalance affects performance isolated from all other factors.