SAFAL: A MapReduce Spatio-temporal Analyzer for UNAVCO FTP Logs

UNAVCO is a National Science Foundation (NSF) funded consortium that facilitates geoscience research and education using geodesy. It is responsible for the collection, archiving and distribution of data from GPS sites installed in every continent of the world. In addition to GPS data, UNAVCO collects borehole seismic, strain meter, meteorological, and digital imagery data. One of UNAVCO´s largest programs is the Plate Boundary Observatory (PBO), the geodetic component of the NSF funded Earth scope program. PBO consists of over 1100 continuous GPS sites plus 80 borehole strain and seismic sites. In this paper, we present SAFAL, a Spatio-temporal Analyzer of FTP Access Logs collected by UNAVCO´s data center. We developed SAFAL using Hadoop/MapReduce. The motivation for this work was to build an efficient system able to quickly identify trends in GPS data usage. The system is able to processes millions of lines of data in minutes. It supports queries such as: (i) what is the most downloaded GPS site, (ii) who is downloading the data most, or (iii) what periods of data are of greatest interest. Answers to these and similar queries are useful for planning network growth, allocating Web resources, and tracking hot topics in geoscience research. They also may be extremely useful to help UNAVCO illuminate dark data.