On Accelerated Gradient Approximation for Least Square Regression with L1-Regularization

In this paper, we consider an online least square regression problem where the objective function is composed of a quadratic loss function and an L₁ regularization on model parameter. For each training sample, we propose to approximate the L₁ regularization by a convex function. This results in an overall convex approximation to the original objective function. We apply an efficient accelerated stochastic approximation algorithm to solve the approximation. The developed algorithm does not need to store previous samples which reduces the space complexity. We further prove that the developed algorithm is guaranteed to converge to the global optimum with a convergence rate O (ln n/√n) where n is the number of training samples. The proof is based on a weaker assumption than those applied in similar research work.