Comparison, utility, and partition of dependence under absolutely continuous and singular distributions

This paper first illustrates that a mutual information index detects and ranks dependence of a wide variety of absolutely continuous families, but the popular association and variance reduction indices fail to serve as such “common metrics”. We then elaborate on some theoretical merits of the mutual information and give several results. The mutual information provides a notion of the utility of dependence for predicting random variables and quantifies how much the joint distribution is more informative about the variables than the independent model. We present insightful partitions of dependence among the components of a random vector, for a class of models recently proposed for dependence of uncorrelated variables, and for the elliptical families. We also recall that the mutual information is not applicable to singular distributions and give some results for a generalized information index for these models. The generalized index is derived for the Marshall–Olkin copula and for a new singular copula that represents the dependence of the consecutive terms of the exponential autoregressive and related processes.