Distributed multistage alert correlation architecture based on Hadoop

There are three main approaches to design when implementing an alert correlation architecture; these are centralised, hierarchical, and decentralised. Centralised approaches benefit from simplicity of implementation and high algorithm expressiveness, but suffer in terms of scalability. The scalability issue is alleviated with hierarchical and decentralised approaches, but this comes at a cost of additional implementation complexity and lower algorithm quality. Introduced is a new alert correlation architecture based on Hadoop. The developed architecture allows for greater scalability whilst maintaining algorithm expressiveness and design simplicity. It incorporates alert aggregation, verification, and correlation components, which together provide for a clear and succinct view of potentially malicious activity. Each component was tested against a series of datasets that represent potential real world scenarios across a cluster of varying size. The results demonstrate that all components in the architecture have the ability to scale across many nodes in a cluster, allowing for the processing of large and complex attack scenarios in a timely manner.