Dynamic Error Recovery in The ATLAS TDAQ System

This paper describes the new dynamic recovery mechanisms in the ATLAS Trigger and Data AcQuisition (TDAQ) system. The purpose is to minimize the impact certain errors and failures have on the system. The new recovery mechanisms are capable of analysing and recovering from a variety of errors, both software and hardware, without stopping the data gathering operations. It incorporates an expert system to perform the analysis of the errors and to decide what measures are needed. Due to the wide array of sub-systems there is also a need to optimize the way similar errors are handled for the different sub-systems. The main focus of the paper is to consider the design and implementation of the new recovery mechanisms and how expert knowledge is gathered from the different sub-systems and implemented in the recovery procedures.