Lightweight Distributed Execution Engine for Large-Scale Spatial Join Query Processing

Existing Big Data systems are mostly designed for relational data. They are either incapable or inefficient in processing large-scale semi-structured data efficiently due to the inherent limitations on data abstraction, indexing support and exposure to native parallel programming tools. In this study, we report our work in developing a lightweight distributed execution engine for spatial join query processing on large-scale geospatial data. By integrating data parallel designs for single computing nodes, our execution engine is able to automatically dispatch data partitions to distributed computing nodes for efficient local execution on multi-core CPUs and GPUs. The execution engine supports asynchronous data transfer over network, asynchronous disk I/O and asynchronous computing. It also directly accesses distributed file systems to support creating and using indices conveniently and efficiently. In addition to be lightweight by design, which has less than 1,000 Lines Of Code (LOC), experiments using a real world application have demonstrated significant efficiency improvement over our previous works on extending a leading in-memory Big Data system (Impala) for spatial join query processing.