Experimental decomposition of the performance of fingerprinting-based localization algorithms

Despite their popularity, the current praxis of comparative experimental evaluation of fingerprinting-based localization algorithms is lacking rigor, with studies typically following an ad-hoc evaluation process and focusing on black-box comparison of complete algorithms. In this paper we present a systematic benchmarking methodology that is focused on gaining finegrained insight about the relative contributions of the individual phases of the fingerprinting-based localization algorithms to their overall performance. To this end, we decompose the localization algorithms in common phases (collection of raw measurements, creation of fingerprints, pattern matching and post-processing) and systematically asses the performance of different procedures that can be applied in each of these phases. We illustrate the application of the proposed methodology using a comprehensive experimental case-study of 3 WiFi fingerprinting algorithms with 4 raw RSSI collection procedures, 3 fingerprint creation and pattern matching procedures, 4 different post-processing procedures in 3 testbeds and 4 evaluation scenarios, resulting in 36 individual experiments. The results demonstrate that in the evaluated scenarios, a lower number of WiFi APs and rather simple fingerprint creation and pattern matching can achieve better performance in terms of location accuracy than more sophisticated alternatives. The results also show that postprocessing steps like k-Nearest Neighbours (kNN) procedure are indeed effective in reducing the localization error variability and extremes, thus increasing the stability of the location estimation.