Hidden dependencies between class imbalance and difficulty of learning for bioinformatics datasets

Many bioinformatics datasets share certain problems: they have class imbalance (one class with many more instances than the remaining class(es)), or are difficult to learn from (build accurate models with). Much research has investigated these two problems, or even considered both at once. However, hidden dependencies can exist between these two problems: in a given collection of datasets, the highly imbalanced datasets may be particularly difficult or easy to learn from, and so conclusions based on the level of class imbalance may actually reflect the difficulty of learning. We present a case study with twenty-six bioinformatics datasets which exhibits this dependency, and highlights how it can result in misleading conclusions regarding the absolute and relative performance of learners and feature rankers across balance levels.