Some worst-case bounds for Bayesian estimators of discrete distributions

Let P be a distribution taking values in a finite set X with cardinality K. Given a sample (X₁, X₂, ..., X_n) a fundamental problem is to estimate the distribution P and its entropy H(P). In practical applications often Bayesian estimators of P and H(P) are used as they may give better results as for example the maximum likelihood estimator. The better performance can be achieved by choosing the right prior distribution on all possible distributions on X. But choosing a wrong prior can be disastrous, especially for entropy estimation, as demonstrated by Nemenman, Shafee, and Bialek in 2001. In this work we give asymptotic worst-case results for Bayesian estimators using a symmetric Dirichlet prior. In particular, we show that estimators using the Laplace and Jeffrey prior can get arbitrarily close to P and H (in L₁ sense) for any distribution P if n scales with K^3/2+δ,δ > 0 as n → ∞. For the Perks and Minimax prior this holds even if n scales with K^1+δ, δ > 0. As a negative result it is further shown that if the Laplace or the Jeffrey prior is used, there is always a distribution P such that the expected L₁ distance is bounded away from zero if n scales linear in K.