The Interplay between Entropy and Variational Distance

For two probability distributions with finite alphabets, a small variational distance between them does not imply that the difference between their entropies is small if one of the alphabet sizes is unknown. This fact, seemingly contradictory to the continuity of entropy for finite alphabet, is clarified in the current paper by means of certain bounds on the entropy difference between two probability distributions in terms of the variational distance between them and their alphabet sizes. These bounds are shown to be the tightest possible. The Lagrange multiplier cannot be applied here because the variational distance is not differentiable. We also show how to find the distribution achieving the minimum (or maximum) entropy among those distributions within a given variational distance from any given distribution. The results show the limitation of certain algorithms for entropy estimation. An upper bound is obtained for the rate-distortion function with respect to the error frequency criterion, and the minimal average complexity is determined for the generation of a probability distribution with a distortion criterion.