Reusing the Results of Queries in MapReduce Systems by Adopting Shared Storage

Zhanye WANG  Chuanyi LIU  Dongsheng WANG  

IEICE TRANSACTIONS on Communications   Vol.E99-B   No.2   pp.315-325
Publication Date: 2016/02/01
Online ISSN: 1745-1345
DOI: 10.1587/transcom.2015ITP0011
Type of Manuscript: Special Section PAPER (Special Section on Management for the Era of Internet of Things and Big Data)
SuperRack,  MapReduce,  results reusing,  shared storage,  

Full Text: PDF>>
Buy this Article

Over the last few years, Apache MapReduce has become the prevailing framework for large scale data processing. Instead of writing MapReduce programs which are too obscure to express, many developers usually adopt high level query languages, such as Hive or Pig Latin, to finish their complex queries. These languages automatically compile each query into a workflow of MapReduce jobs, so they greatly facilitate the querying and management of large datasets. One option to speed up the execution of workflows is to save the results produced previously and reuse them in the future if needed. In this paper we present SuperRack, which uses shared storage devices to store the results of each workflow and allows a new query to reuse these results in order to avoid redundant computation and hasten execution. We propose several novel techniques to improve the access and storage efficiency of the previous results. We also evaluate SuperRack to verify its feasibility and effectiveness. Experiments show that our solution outperforms Hive significantly under TPC-H benchmark and real life workloads.