A New Efficient Resource Management Framework for Iterative MapReduce Processing in Large-Scale Data Analysis

Seungtae HONG  Kyongseok PARK  Chae-Deok LIM  Jae-Woo CHANG  

IEICE TRANSACTIONS on Information and Systems   Vol.E100-D   No.4   pp.704-717
Publication Date: 2017/04/01
Online ISSN: 1745-1361
Type of Manuscript: Special Section PAPER (Special Section on Data Engineering and Information Management)
large-scale data analysis,  iterative data processing framework,  MapReduce,  Hadoop,  

Full Text: PDF(3.6MB)
>>Buy this Article | Errata[Uploaded on March 1,2018]

To analyze large-scale data efficiently, studies on Hadoop, one of the most popular MapReduce frameworks, have been actively done. Meanwhile, most of the large-scale data analysis applications, e.g., data clustering, are required to do the same map and reduce functions repeatedly. However, Hadoop cannot provide an optimal performance for iterative MapReduce jobs because it derives a result by doing one phase of map and reduce functions. To solve the problems, in this paper, we propose a new efficient resource management framework for iterative MapReduce processing in large-scale data analysis. For this, we first design an iterative job state-machine for managing the iterative MapReduce jobs. Secondly, we propose an invariant data caching mechanism for reducing the I/O costs of data accesses. Thirdly, we propose an iterative resource management technique for efficiently managing the resources of a Hadoop cluster. Fourthly, we devise a stop condition check mechanism for preventing unnecessary computation. Finally, we show the performance superiority of the proposed framework by comparing it with the existing frameworks.