Minimax rates of convergence for high-dimensional regression under ℓq-ball sparsity

Consider the standard linear regression model y = Xß* + w, where y ¿ Rⁿ is an observation vector, X ¿ R^n×d is a measurement matrix, ß* ¿ R^d is the unknown regression vector, and w ~ N (0, ¿² I) is additive Gaussian noise. This paper determines sharp minimax rates of convergence for estimation of ß* in ¿₂ norm, assuming that ß* belongs to a weak ¿_b-ball B_q(R_q) for some q ¿ [0, 1]. We show that under suitable regularity conditions on the design matrix X, the minimax error in squared ¿₂-norm scales as R_q((log d)/n)^{1 -q/2}. In addition, we provide lower bounds on rates of convergence for general ¿_p norm (for all p ¿ [1, + ¿], p ¿ q). Our proofs of the lower bounds are information-theoretic in nature, based on Fano´s inequality and results on the metric entropy of the balls B_q(R_q). Matching upper bounds are derived by direct analysis of the solution to an optimization algorithm over B_q(R_q). We prove that the conditions on X required by optimal algorithms are satisfied with high probability by broad classes of non-i.i.d. Gaussian random matrices, for which RIP or other sparse eigenvalue conditions are violated. For q = 0, ¿₁-based methods (Lasso and Dantzig selector) achieve the minimax optimal rates in ¿₂ error, but require stronger regularity conditions on the design than the non-convex optimization algorithm used to determine the minimax upper bounds.