Least squares estimates and the coverage of least squares costs

The least squares estimate x̂_N minimizes the sum of the squared residuals equation over a finite set of observations (A_i, b_i). At x = x̂_N, the squared residuals ∥A_ix̂_N-b_i∥² are called the “empirical costs”. Intuitively, the empirical costs carry information on the probability distribution of the cost ∥Ax̂_N-b∥² that is paid for other, yet unseen, values of (A, b) taken from the same population as the observations (A_i, b_i). In this work, this intuition is set on solid theoretical grounds. We provide a precise characterization of the probabilities with which the cost does not exceed certain thresholds that are constructed from the empirical costs. These probabilities are called “coverages”. All the results are derived in a setting where the observations are independent, while the framework is otherwise “agnostic” in that no a-priori assumptions about the underlying probability for (A, b) is made.