Experiences and Challenges in Enhancing Security Information and Event Management Capability Using Unsupervised Anomaly Detection

Security Information and Event Management (SIEM) systems are important components of security and threat management in enterprises. To compensate for the shortcomings of rule-based correlation in this field, there has been an increasing demand for advanced anomaly detection techniques. Such implementations, where prior training data is not required, have been described previously. In this paper, we focus on the requirements for such a system and provide insight into how diverse security events need to be parsed, unified and preprocessed to meet the requirements of unsupervised anomaly detection algorithms. Specific focus is given to the detection of suspicious authentication attempts, password guessing attacks and unusual user account activities in a large-scale Microsoft Windows domain network. In the course of this paper we analyze a comprehensive dataset of 15 million Windows security events from various perspectives using the k-nearest neighbor algorithm. Key considerations on how to effectively apply anomaly detection are proposed in order to produce accurate and convincing results. The effectiveness of our approach is discussed using sample anomalies that were detected in the analyzed data.