Scalable cluster administration - Chiba City I approach and lessons learned

Systems administrators of large clusters often need to perform the same administrative task hundreds or thousands of times. Administrators have traditionally performed some time-consuming tasks, such as operating system installation, configuration, and maintenance, manually. By combining network services such as DHCP, TFTP, FTP, HTTP, and NFS with remote hardware control and scripted installation, configuration, and maintenance techniques, cluster administrators can automate these administrative tasks. Scalable cluster administration addresses this challenge: What hardware and software design techniques can cluster builders use to automate cluster administration on very large clusters? We describe the approach used in the Mathematics and Computer Science Division of Argonne National Laboratory on Chiba City I, a 314-node Linux cluster; and we analyze the scalability, flexibility, performance and reliability benefits and limitations from that approach.