Lossy Source Coding via Markov Chain Monte Carlo

We propose an implementable new universal lossy source coding algorithm. The new algorithm utilizes two well-known tools from statistical physics and computer science: Gibbs sampling and simulated annealing. In order to code a source sequence xⁿ, the encoder initializes the reconstruction block as x circⁿ = xⁿ, and then at each iteration uniformly at random chooses one of the symbols of x circⁿ, and updates it. This updating is based on some conditional probability distribution which depends on a parameter beta representing inverse temperature, an integer parameter k = o(log n) representing context length, and the original source sequence. At the end of this process, the encoder outputs the Lempel-Ziv description of x circⁿ, which the decoder deciphers perfectly, and sets as its reconstruction. The complexity of the proposed algorithm in each iteration is linear in k and independent of n. We prove that, for any stationary ergodic source, the algorithm achieves the optimal rate-distortion performance asymptotically in the limits of large number of iterations, beta, and n.