Optimizing MapReduce scheduling using datanode load prediction

MapReduce [3] is a leading distributed programming model for data-intensive computing. The strength of the MapReduce program depends on splitting the large datasets into the small input blocks and processes them on the cluster of machines. The Optimization of the data-intensive computing model is mainly relying on the Job scheduler [5] component of Hadoop MapReduce framework. The JobTracker assigns the job to the TaskTrackers with the help of job scheduler [14]. The JobTracker does not consider the CPU intensive task running on the TaskTrackers while assigning new task to the TaskTrackers which leads to the node crash or failure. In our proposed Research method, job Tracker schedules the job by considering the load statistics of the TaskTracker into the account. TaskTracker will send current load statistics by modifying heartbeat message. The load statistic information includes the CPU information, physical memory, swap memory, disk IO etc.