Empirical Information Metrics for Prediction Power and Experiment Planning

In principle, information theory could provide useful metrics for statisticalinference. In practice this is impeded by divergent assumptions: Information theoryassumes the joint distribution of variables of interest is known, whereas in statisticalinference it is hidden and is the goal of inference. To integrate these approaches wenote a common theme they share, namely the measurement of prediction power. Wegeneralize this concept as an information metric, subject to several requirements: Calculationof the metric must be objective or model-free; unbiased; convergent; probabilisticallybounded; and low in computational complexity. Unfortunately, widely used model selectionmetrics such as Maximum Likelihood, the Akaike Information Criterion and BayesianInformation Criterion do not necessarily meet all these requirements. We define four distinctempirical information metrics measured via sampling, with explicit Law of Large Numbersconvergence guarantees, which meet these requirements: Ie, the empirical information, ameasure of average prediction power; Ib, the overfitting bias information, which measuresselection bias in the modeling procedure; Ip, the potential information, which measures thetotal remaining information in the observations not yet discovered by the model; and Im, themodel information, which measures the model’s extrapolation prediction power. Finally, weshow that Ip + Ie, Ip + Im, and Ie 􀀀 Im are fixed constants for a given observed dataset (i.e. prediction target), independent of the model, and thus represent a fundamental subdivisionof the total information contained in the observations. We discuss the application of thesemetrics to modeling and experiment planning