On the Application and Performance of MongoDB for Climate Satellite Data

Analyses in climate research typically operate on large datasets stored in file hierarchies. However, e.g. Indexing, meta-data search and replication is challenging with this approach. A new approach is to store and index datasets in large, distributed databases. In order to demonstrate the performance improvement, we utilized the so-called general matching problem between measurements of two satellites that differ in orbits and measurement cycles. For comparison purpose the measurements are matched within a specified maximum spatial and time offset constraint. We describe the steps from a single-threaded approach using a SQL database to a multi-threaded approach using the NoSQL database MongoDB. The performance as well as limitations on CPU and I/O are evaluated and discussed for each approach. The performance has been improved up to a factor of 46 using 11 threads on a 12-core system.