Improving Data Processing Time with Access Sequence Prediction

Genomic research nowadays often faces the problem of big data. The data size from genome sequencing process can grow very quickly and continuously creating the problem with storage and processing. BGI, one of the renowned genomic research institutes in China also faces the similar problem. The research at BGI depends on several sequencing machines. One machine pipeline may generate temporary data of around 1.4 terabytes. In addition, multiple read and write operations occur continuously during processing time. The I/O bottleneck thus degrades research throughput tremendously. Using a high performance computing system alone is not sufficiently effective in experimental results processing. In order to hide the I/O latency, an effective big data management framework is needed at BGI. In this paper, we proposed the hybrid prediction model for data access pattern. The goal is to predict the next pieces of data needed in the processor and preload them into the memory in order to improve the overall processing time. From the results obtained from the initial experiments, the proposed model can deliver high prediction accuracy in linear-time. Moreover, the error rate is low at 1.85%, which is better than the common methods used, such as Prediction Graph, ANN and ARMA. We believe that with some further fine-tuning, the model can be used as a part of the big data management framework deployed at BGI in the near future.