On redundancy of memoryless sources over countable alphabets

In this paper, we study the average redundancy incurred by the universal description of strings of positive integers (Z₊), the strings being generated independently and identically distributed (i.i.d.) according an unknown distribution p over Z₊ in a known collection P. The redundancy of describing a single symbol is a crucial metric here-if description of a single symbol incurs finite redundancy then the class P is weakly compressible. Namely there is a universal measure q over infinite strings of positive integers such that for all p ∈ P, the normalized redundancy of describing length-n strings diminishes to 0 as n n → ∞. We show that if redundancy of describing a single symbol is finite, the collection P is tight but that the converse does not always hold. In contrast to worst case formulations, we show that it is possible that the description of a single symbol from an unknown distribution of P incurs finite average redundancy, yet the description of length n i.i.d. strings incurs a constant (> 0) redundancy per symbol encoded. We then show a sufficient condition on single-letter marginals, such that length n i.i.d. samples will incur vanishing redundancy per symbol encoded.