Developing a Real-Time Data Analytics Framework Using Hadoop

Currently, the majority of existing workflows are based on meta-heuristics that produce good heuristics that are dynamic in nature, and map the workflow tasks to services on-the-fly, but unfortunately, they lack the ability of supporting analytical tasks considering data types and real-time processing. This paper aims to address this problem by developing a real-time data analytics framework capable of handling real-time processing of structured and unstructured data needed for performing different analytical tasks, ranging from data ingestion and processing to data exploration, and visualization. We propose architecture based on the Storm/YARN projects for data ingestion, processing exploration and visualization of streaming structured and unstructured data. We have implemented the proposed architecture using Apache Storm related APIs for both of a local mode and a distributed mode. We describe our experiments for the architecture prototype implementation and evaluate the functional requirements for each component and non-functional tests such as real time update performance and time taken for data flow among components. All components were able to handle their own functionalities properly. Also, we provide the main results for a non-functional test in order to discuss our system efficiency.