System Status Aware Hadoop Scheduling Methods for Job Performance Improvement


IEICE TRANSACTIONS on Information and Systems   Vol.E98-D   No.7   pp.1275-1285
Publication Date: 2015/07/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2014EDP7385
Type of Manuscript: PAPER
Category: Fundamentals of Information Systems
Hadoop,  MapReduce,  distributed computing,  task scheduling,  job performance,  

Full Text: PDF(3.8MB)>>
Buy this Article

MapReduce and its open software implementation Hadoop are now widely deployed for big data analysis. As MapReduce runs over a cluster of massive machines, data transfer often becomes a bottleneck in job processing. In this paper, we explore the influence of data transfer to job processing performance and analyze the mechanism of job performance deterioration caused by data transfer oriented congestion at disk I/O and/or network I/O. Based on this analysis, we update Hadoop's Heartbeat messages to contain the real time system status for each machine, like disk I/O and link usage rate. This enhancement makes Hadoop's scheduler be aware of each machine's workload and make more accurate decision of scheduling. The experiment has been done to evaluate the effectiveness of enhanced scheduling methods and discussions are provided to compare the several proposed scheduling policies.