GEM: A Framework for Developing Shared-Memory Parallel Genomic Applications on Memory Constrained Architectures

Amount of available genomic data is increasing rapidly with the recent developments in sequencing technologies. Analysis of such data can potentially lead significant advancements in medical research and even practice. However, it is imperative to exploit parallelism and utilize computational resources effectively to handle large scale genomic data. At the same time, the trends in computing technologies are towards architectures with large number of cores and smaller memory size per core (e.g. Intel Xeon Phi). Innovative solutions that meet the requirements of parallel genomic data processing with the constraints of the new computational architectures are urgently needed. In this work, we develop a novel middleware system, GEM, for developing shared-memory parallel genomic applications with memory constraint architectures. We propose load-map-reduce approach and a novel scheduling scheme to decrease I/O contention and prevent over-consumption of the limited memory. We also use domain specific knowledge to decrease the memory requirements of the tasks. In our experiments, we show that GEM has high scalability on Intel Xeon Phi architecture. We also compare GEM against two other frameworks for genomic data processing, GATK and PAGE, and show that our middleware outperforms both.