Runtime Composition for Extensible Big Data Processing Platforms

We propose a runtime composition method for creating various applications that comprise elaborate big data processing. Cloud platforms are required to be more capable at applying elaborate big data processing to various services for real businesses. Therefore, the platforms should be extensible to compound emerging runtimes. However, there are two major problems with this extensibility: providing an easy method for compounding a new runtime to the platforms and how to select the most efficient runtimes for each process in various situations. The runtime composition method is based on the model-driven engineering approach. The method searches the types of the most cost-efficient runtimes for each process under the requirements, creates the intermediate model, and, finally, generates files deployed to different processing engines. We designed and implemented four algorithms for selecting the runtimes. To confirm the ease of extending the method, we also designed and implemented four types of plugins. We found that we can separately append each type of information by installing these plugins. Experimental results show that variable depth search is the most preferable because of the accuracy of execution cost, but the greedy algorithm may be also preferable because of the balance among the execution time, the error rate, and the ease of implementation.