pLSM: A Highly Efficient LSM-Tree Index Supporting Real-Time Big Data Analysis

Big Data boosts the development of data management and analysis in database systems but it also poses a challenge to traditional database. NoSQL databases are provided to deal with the new challenges brought by Big Data because of its high performance, storage, scalability and availability. In NoSQL databases, it is an essential requirement to provide scalable and efficient index services for real-time data analysis. Most existing index solutions focus on improving write throughput, but at the cost of poor read performance. We designed a new plug-in system PuntStore with pLSM (Punt Log Structured Merge Tree) index engine. To improve read performance, Cache Oblivious Look-ahead Array (COLA) is adopted in our design. We also presented a novel compact algorithm in bulk deletion to support migration of data from temporary storage to data warehouse for further analysis.