Using Confusion Matrices to Estimate Mutual Information between Two Categorical Measurements

Many data analysis problems in neuroimaging are set up as classification problems, frequently involving multiple classes. Typically, only the fraction of correct classifications is reported as aggregate accuracy. However, the structure of the classification errors contains valuable information as well. By reinterpreting confusion matrices as conditional probabilities we not only demonstrate a procedure for computing the mutual information between a categorical measurement and ground truth, but we also derive a mechanism for computing mutual information between two separate measurements of the same shared ground truth. We demonstrate this approach with fMRI and behavioral data for categorization of natural scenes.