Minimax Rates of Estimation for High-Dimensional Linear Regression Over $\ell _q$ -Balls

Consider the high-dimensional linear regression model y = X β^* + w, where y ∈ BBRn is an observation vector, X ∈ BBRn × d is a design matrix with d >; n, β^* ∈ BBRd is an unknown regression vector, and w ~ N(0, σ²I) is additive Gaussian noise. This paper studies the minimax rates of convergence for estimating β^* in either l₂-loss and l₂-prediction loss, assuming that β^* belongs to an lq -ball BBBq(Rq) for some q ∈ [0,1]. It is shown that under suitable regularity conditions on the design matrix X, the minimax optimal rate in l₂-loss and l₂-prediction loss scales as Θ(Rq ([(logd)/(n)])^1-q/₂). The analysis in this paper reveals that conditions on the design matrix X enter into the rates for l₂-error and l₂-prediction error in complementary ways in the upper and lower bounds. Our proofs of the lower bounds are information theoretic in nature, based on Fano´s inequality and results on the metric entropy of the balls BBBq(Rq), whereas our proofs of the upper bounds are constructive, involving direct analysis of least squares over lq-balls. For the special case q=0, corresponding to models with an exact sparsity constraint, our results show that although computationally efficient l₁-based methods can achieve the minimax rates up to constant factors, they require slightly stronger assumptions on the design matrix X than optimal algorithms involving least-squares over the l₀-ball.