Managing the Synchronization in the Lambda Architecture for Optimized Big Data Analysis

Thomas VANHOVE  Gregory VAN SEGHBROECK  Tim WAUTERS  Bruno VOLCKAERT  Filip DE TURCK  

Publication
IEICE TRANSACTIONS on Communications   Vol.E99-B   No.2   pp.297-306
Publication Date: 2016/02/01
Online ISSN: 1745-1345
Type of Manuscript: INVITED PAPER (Special Section on Management for the Era of Internet of Things and Big Data)
Category: 
Keyword: 
Lambda architecture,  synchronization,  big data,  Tengu,  

Full Text: FreePDF(2MB)


Summary: 
In a world of continuously expanding amounts of data, retrieving interesting information from enormous data sets becomes more complex every day. Solutions for precomputing views on these big data sets mostly follow either an offline approach, which is slow but can take into account the entire data set, or a streaming approach, which is fast but only relies on the latest data entries. A hybrid solution was introduced through the Lambda architecture concept. It combines both offline and streaming approaches by analyzing data in a fast speed layer first, and in a slower batch layer later. However, this introduces a new synchronization challenge: once the data is analyzed by the batch layer, the corresponding information needs to be removed in the speed layer without introducing redundancy or loss of data. In this paper we propose a new approach to implement the Lambda architecture concept independent of the technologies used for offline and stream computing. A universal solution is provided to manage the complex synchronization introduced by the Lambda architecture and techniques to provide fault tolerance. The proposed solution is evaluated by means of detailed experimental results.