DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce

Zhihong LIU  Aimal KHAN  Peixin CHEN  Yaping LIU  Zhenghu GONG  

IEICE TRANSACTIONS on Information and Systems   Vol.E99-D   No.6   pp.1686-1689
Publication Date: 2016/06/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2015EDL8255
Type of Manuscript: LETTER
Category: Fundamentals of Information Systems
MapReduce,  resource-aware scheduling,  skew mitigation,  

Full Text: PDF(667.3KB)>>
Buy this Article

MapReduce still suffers from a problem known as skew, where load is unevenly distributed among tasks. Existing solutions follow a similar pattern that estimates the load of each task and then rebalances the load among tasks. However, these solutions often incur heavy overhead due to the load estimation and rebalancing. In this paper, we present DynamicAdjust, a dynamic resource adjustment technique for mitigating skew in MapReduce. Instead of rebalancing the load among tasks, DynamicAdjust adjusts resources dynamically for the tasks that need more computation, thereby accelerating these tasks. Through experiments using real MapReduce workloads on a 21-node Hadoop cluster, we show that DynamicAdjust can effectively mitigate the skew and speed up the job completion time by up to 37.27% compared to the native Hadoop YARN.