Faster SVD-truncated regularized least-squares

We develop a fast algorithm for computing the “SVD-truncated” regularized solution to the least-squares problem: min_x ∥Ax - b∥₂. Let A_k of rank k be the best rank k matrix computed via the SVD of A. Then, the SVD-truncated regularized solution is: x_k = A_k^†b. If A is m × n, then, it takes O(mnmin{m, n}) time to compute x_k using the SVD of A. We give an approximation algorithm for x_k which constructs a rank k approximation Ã_k and computes x̃_k = Ã_k^† in roughly O(nnz(A)k log n) time. Our algorithm uses a randomized variant of the subspace iteration method. We show that, with high probability: ∥Ax̃_k - b∥₂ ≈ ∥Ax_k - b∥₂ and ∥x_k - x̃_k∥₂ ≈ 0.