A Compatible LZMA ORC-Based Optimization for High Performance Big Data Load

This paper presents several efficient ways to improve data loading and storage optimization in Hadoop cluster. We design a new method to leverage LZMA and ORC to gain performance edge, also improve ORC implementation in HDFS to have a higher compression ratio and better IO throughput. A complete optimization strategy for efficient big data loading, including byte array-oriented, record split, less serialization and shuffle, reducing middle data landing to earn great performance boost is presented. This paper provides preliminary results and analytics. Evaluation results indicate that our method achieves significant performance improvement for big data load.