DIRAC: a scalable lightweight architecture for high throughput computing

DIRAC (distributed infrastructure with remote agent control) has been developed by the CERN LHCb physics experiment to facilitate large scale simulation and user analysis tasks spread across both grid and nongrid computing resources. It consists of a small set of distributed stateless core services, which are centrally managed, and agents which are managed by each computing site. DIRAC utilizes concepts from existing distributed computing models to provide a lightweight, robust, and flexible system. This paper discusses the architecture, performance, and implementation of the DIRAC system which has recently been used for an intensive physics simulation involving more than forty sites, 90 TB of data, and in excess of one thousand 1 GHz processor-years.