Utility of Dependence : Reduction of Uncertainty and Departure from Independence

Utility of Dependence : Reduction of Uncertainty and Departure from Independence

Dependence between variables with a multivariate normal distribution can easily be assessed through the correlation measures. But in general dependence is more complicated than that could be measured by the traditional indices such as the correlation coefficients, its nonparametric counterparts, and the fraction of variance reduction. The mutual information, denoted here as M. measures departure of a joint distribution from the independent model. We also view M as an expected utility of variables for prediction. This view integrates ideas from the general dependence literature and the Bayesian information. After an overview of its theoretical foundations, we show that M provides a robust dependence index which extends the interpretations of the normal squared correlation to all distributions that are absolutely continuous relative to the product of their marginals. We illustrate the success of this index as a “common metric for comparing the strength of dependence within and between families of distributions in contrast with the failures of the popular traditional indices. For the location-scale family of distributions, an additive decomposition of AI gives the normal distribution as the unique minimal dependence model in the family. An implication for practice is that the popular association indices underestimate the dependence of elliptical distributions, severely for models such as t distributions with low degrees of freedom. Finally, we draw attention to a caveat: M is not applicable to continuous variables when their joint distribution is singular.