Error log analysis: statistical modeling and heuristic trend analysis

Most error-log analysis studies perform a statistical fit to the data assuming a single underlying error process. The authors present the results of an analysis that demonstrates that the log is composed of at least two error processes: transient and intermittent. The mixing of data from multiple processes requires many more events to verify a hypotheses using traditional statistical analysis. Based on the shape of the interarrival time function of the intermittent errors observed from actual error logs, a failure-prediction heuristic, the dispersion frame technique (DFT), is developed. The DFT was implemented in a distributed system for the campus-wide Andrew file system at Carnegie Mellon University. Data collected from 13 file servers over a 22-month period were analyzed using both the DFT and conventional statistical methods. It is shown that the DFT can extract intermittent errors from the error log and uses only one fifth of the error-log entry points required by statistical methods for failure prediction. The DFT achieved a 93.7% success rate in predicting failures in both electromechanical and electronic devices