Dynamic Error Recovery in theATLAS TDAQ System

This paper describes the new dynamic recovery mechanisms in the ATLAS Trigger and DataAcQuisition (TDAQ) system. The purpose of the new recovery mechanism is to minimize the impact certain errors and failures have on the system. The new recovery mechanisms are capable of analyzing and recovering from a variety of errors, both software and hardware, without stopping the data-gathering operations. An expert system is incorporated to perform the analysis of the errors and to decide what measures are needed. Due to the wide array of sub-systems there is also a need to optimize the way similar errors are handled for the different sub-systems. The main focus of the paper is to consider the design and implementation of the new recovery mechanisms and how expert knowledge is gathered from the different sub-systems and implemented in the recovery procedures.