Performance of Splunk for the TDAQ information service at the ATLAS experiment

The ATLAS Trigger and Data Acquisition (TDAQ) is a large, distributed system composed of several thousand interconnected computers and tens of thousands software processes. Monitoring data produced by multiple sources are selected, aggregated and correlated to perform the analysis of the monitored data. Then they can finally be visualized and presented to the user. Any system implementing these functions has to be flexible in order to adapt to the amount of data produced and requested by the users for analysis and visualization. Due to the size of the ATLAS TDAQ system, the scalability is also important from the performance point of view. Splunk, a commercial product produced by Splunk Inc., is a general-purpose search, analysis & reporting engine and a distributed, non-relational, semi-structured database for time-series text data. This paper describes the evaluation of Splunk for the functionality and the performance.