The source coding theorem revisited: A combinatorial approach

A combinatorial approach is proposed for proving the classical source coding theorems for a finite memoryless stationary source (giving achievable rates and the error probability exponent). This approach provides a sound heuristic justification for the widespread appearence of entropy and divergence (Kullback\´s discrimination) in source coding. The results are based on the notion of composition class -- a set made up of all the distinct source sequences of a given length which are permutations of one another. The asymptotic growth rate of any composition class is precisely an entropy. For a finite memoryless constant source all members of a composition class have equal probability; the probability of any given class therefore is equal to the number of sequences in the class times the probability of an individual sequence in the class. The number of different composition classes is algebraic in block length, whereas the probability of a composition class is exponential, and the probability exponent is a divergence. Thus if a codeword is assigned to all sequences whose composition classes have rate less than some rate , the probability of error is asymptotically the probability of the must probable composition class of rate greater than . This is expressed in terms of a divergence. No use is made either of the law of large numbers or of Chebyshev\´s inequality.