On competitive prediction and its relation to rate-distortion theory

Consider the normalized cumulative loss of a predictor F on the sequence xⁿ=(x₁,...,x_n), denoted L_F(xⁿ). For a set of predictors G, let L(G,xⁿ)=min_F∈GL_F(xⁿ) denote the loss of the best predictor in the class on xⁿ. Given the stochastic process X=X₁,X₂,..., we look at EL(G,Xⁿ), termed the competitive predictability of G on Xⁿ. Our interest is in the optimal predictor set of size M, i.e., the predictor set achieving min_|G|≤MEL(G,Xⁿ). When M is subexponential in n, simple arguments show that min_|G|≤MEL(G,Xⁿ) coincides, for large n, with the Bayesian envelope min_FEL_F(Xⁿ). We investigate the behavior, for large n, of min_|G|≤e^nREL(G,Xⁿ), which we term the competitive predictability of X at rate R. We show that whenever X has an autoregressive representation via a predictor with an associated independent and identically distributed (i.i.d.) innovation process, its competitive predictability is given by the distortion-rate function of that innovation process. Indeed, it will be argued that by viewing G as a rate-distortion codebook and the predictors in it as codewords allowed to base the reconstruction of each symbol on the past unquantized symbols, the result can be considered as the source-coding analog of Shannon´s classical result that feedback does not increase the capacity of a memoryless channel. For a general process X, we show that the competitive predictability is lower-bounded by the Shannon lower bound (SLB) on the distortion-rate function of X and upper-bounded by the distortion-rate function of any (not necessarily memoryless) innovation process through which the process X has an autoregressive representation. Thus, the competitive predictability is also precisely characterized whenever X can be autoregressively represented via an innovation process for which the SLB is tight. The error exponent, i.e., the exponential behavior of min_{|G|≤exp(nR)}Pr(L(G,Xⁿ)>d), is also characterized for processes that can be autoregressively represented with an i.i.d. innovation process.