A data reusing strategy based on hive

Large scale data process has emerged as an important issue for concerned researchers. By reusing calculation results, the efficiency of large scale data process can be improved greatly. This paper proposes an efficient data reusing strategy based on the data warehouse tool-Hive, which works on MapReduce framework. Since the intermediate calculation results have been stored in DFS by different jobs in MapReduce workflow, the key issue is how to find the ruse information. This paper deals with this problem by two steps. In the proposed method, firstly, we define a joint object to organize and store the features of intermediate calculation results. Then, based on joint objects, this paper provides the algorithm to match and generate the reuse plan. This paper provides a way to obtain the best reuse strategy in case that there are more than one calculation result can be used. We conduct the experiments based on TPC-H and SSB benchmarks. The experimental results have demonstrated that our strategy can significantly improve the efficiency of large scale data process, and have little effect on queries executed at first time.