An automatic approach to extract the formats of network and security log messages

Analyzing massive network and security logs that record network events is crucial for diagnosing network anomalies in large-scale network environments. Extracting log message formats is an important and necessary step to achieve the goal. However, it is time-consuming and costly to automatically and efficiently extract log message formats from massive network and security logs of many different types, which are generated by the increasing number of network and security devices and services used in large-scale networks. In this paper, we propose log template extraction (LTE), an approach that is semantics aware of network and security logs to address the problem. LTE first cleans log messages and then clusters the cleaned log messages based on the DBSCAN algorithm. At last it infers message templates by LDA Gibbs sampling algorithm. We evaluate our work on massive amount of network log messages collected from a large production network. Experimental results show that LTE approach infers and gets multiple log message formats at the same time with more than 90% accuracy and 100% recall.