Dominant Fairness Fairness: Hierarchical Scheduling for Multiple Resources in Heterogeneous Datacenters

Wenzhu WANG  Kun JIANG  Yusong TAN  Qingbo WU  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E99-D   No.6   pp.1678-1681
Publication Date: 2016/06/01
Publicized: 2016/03/03
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2015EDL8253
Type of Manuscript: LETTER
Category: Fundamentals of Information Systems
Keyword: 
hierarchical scheduling,  heterogeneous datacenter,  multiple resources,  H-DRF,  

Full Text: PDF(225.2KB)>>
Buy this Article




Summary: 
Hierarchical scheduling for multiple resources is partially responsible for the performance achievements in large scale datacenters. However, the latest scheduling technique, Hierarchy Dominant Resource Fairness (H-DRF)[1], has some shortcomings in heterogeneous environments, such as starving certain jobs or unfair resource allocation. This is because a heterogeneous environment brings new challenges. In this paper, we propose a novel scheduling algorithm called Dominant Fairness Fairness (DFF). DFF tries to keep resource allocation fair, avoid job starvation, and improve system resource utilization. We implement DFF in the YARN system, a most commonly used scheduler for large scale clusters. The experimental results show that our proposed algorithm leads to higher resource utilization and better throughput than H-DRF.