Big Data Processing Systems: State-of-the-Art and Open Challenges

The growing demand for large-scale data processing and data analysis applications spurred the development of novel solutions from both the industry and academia. In the last decade, the MapReduce framework has emerged as a highly successful framework that has created a lot of momentum in both the research and industrial communities such that it has become the defacto standard of big data processing platforms. In particular, the MapReduce framework has been introduced to provide a simple but powerful programming model and runtime environment that eases the job of developing scalable parallel applications to process vast amounts of data on large clusters of commodity machines. However, recently, academia and industry have started to recognize the limitations of the Hadoop framework in several application domains such as large scale processing of structured data, graph data and streaming data. Thus, in recent years, we have witnessed an unprecedented interest to tackle these challenges which constitutes a new wave of domain-specific optimized big data processing platforms. To better understand the latest ongoing developments in the world of big data processing systems, in this paper, we provide a detailed overview and analysis of the state-of-the-art in this domain. In addition, we identify a set of the current open research challenges and discuss some promising directions for future research.