Distributed and parallel random coordinate descent methods for huge convex programming over networks

In this paper we develop parallel random coordinate gradient descent methods for minimizing huge linearly constrained separable convex problems over networks. Since we have coupled constraints in the problem, we devise a family of algorithms that updates in parallel τ ≥ 2 (block) components per iteration. Moreover, the algorithms are adequate for distributed and parallel computations and their complexity per iteration is cheaper than of the full gradient method when the number of nodes N in the network is huge. We prove that for these methods we obtain in expectation an o-accurate solution in at most O(N over τε) iterations and thus the convergence rate depends linearly on the number of (block) components to be updated. We also describe several applications that fit in our framework, in particular the convex feasibility problem. Numerically, we show that the parallel coordinate descent method with τ > 2 accelerates on its basic counterpart corresponding to τ = 2.