Fault–Tolerant Master-Workers Framework for MapReduce Applications

Over the past few years, the top web companies like Google, Yahoo etc have come up with several special techniques that process large amounts of raw data, such as crawled documents, web request logs, etc. The issue is how to handle large computations and large data. Parallel computing is the remedy for reducing the response time. The challenge here is to parallelizing the computation part and then Google introduced a technique called MapReduce to deal with this challenge. In this paper we are proposing a general Master-Workers (MW) Framework i.e., a framework for increasing the performance of the MapReduce applications. With various kinds of experiments we have shown our parallel framework which is efficient and robust for executing MapReduce applications on clusters. To this end, we build MW Framework for the users that can efficiently execute any MapReduce applications by varying parameters such as data size and the number of workers available. Our MW Framework is also robust against faults and poor machines.