Evaluating the consistency of estimation

The error covariance reported by an estimation is said to be consistent if it is a reliable indicator of the actual error. In this paper several types of consistency are defined, and methods for its evaluation are introduced. Mean Squared Deviation consistency is based on the Chebyshev inequality, p equivalence is based on the fact that the concentration ellipse with probability mass p must contain the actual value of the estimated parameter with probability p, and Normalized Deviation Squared (NDS) consistency implies that a concentration ellipse of probability mass p contains the actual value of the estimated parameter with probability at least p. Hypothesis tests for consistency evaluation are presented. The NDS consistency test is applied to WiFi localization system data in order to investigate sources of inconsistencies and adjust parameters of the system. It is shown that underestimated measurement noise is the main cause of inconsistent behavior; however, an incorrect motion model or underestimated process noise might also result in inconsistent estimates.