Lossy source coding with sparse graph codes: A variational formulation of soft decimation

Various authors have obtained state-of-the-art results in lossy source coding by applying algorithms based on a combination of message-passing and decimation to low-density generator matrix codes, but to date, theoretical understanding of these procedures has been limited. We show that certain forms of soft decimation can be understood as iterative procedures for attempting to maximize a cost function of the node biases. This variational characterization allows us to exhibit appropriate choices of stepsize that ensure convergence to a fixed point, and to provide guarantees on the distortion of the encoding obtained from the fixed point for the case of symmetric Bernoulli sources. Our analysis applies to both an oracle form of soft decimation, in which exact marginals can be computed, and a practical form based on the (reweighted) sum-product algorithm.