Degree-guided map-reduce task assignment with data locality constraint

The map-reduce framework is used in many data-intensive parallel processing systems. Data locality is an important problem with map-reduce as tasks with local data complete faster than those with remote data. We propose a degree-guided task assignment algorithm, which uses very little extra information than the currently implemented Random Server algorithm. We analyze a simple version of the degree-guided algorithm, called Peeling algorithm, and the Random Server algorithm in a discrete-time model using evolution of random graphs. We characterize the thresholds below which no queueing takes place and compute the effective service rates for both algorithms. The degree-guided algorithm achieves the optimal performance in the region of practical interest and significantly outperforms the Random Server algorithm. The performance characteristics derived from discrete time model are confirmed with simulation in continuous time.