Dynamic learning of automata from the call stack log for anomaly detection

Anomaly detection based on monitoring of sequences of system calls has proved to be an effective approach for detection of previously unknown attacks on programs. This paper describes a new model for profiling normal program behavior that can be used to detect intrusions that change application execution flow. The model (hybrid push down automaton, HPDA) incorporates call stack information and can be learned by dynamic analysis of training data captured from the call stack log. The learning algorithm uses call stack information maintained by the program to build a finite state automaton. When compared to other approaches including VtPath which also uses call stack information, the HPDA model produces a more compact and general representation of control flow, handles recursion naturally, can be learned with less training data, and has a lower false positive rate when used for anomaly detection. In addition, dynamic learning can also be used to supplement a model acquired from static analysis.