Data partitioning and load balancing in parallel storage systems

Summary form only given, as follows. Parallel storage systems such as disk arrays or disk farms provide opportunities for exploiting I/O parallelism in two possible ways: via interrequest parallelism and via intrarequest parallelism. We argue for software-controlled storage systems in which each disk can be accessed individually and data partitioning, as well as data allocation, is completely under the control of the file system. We discuss the main issues in performance tuning of such systems-striping and load balancing-and show their relationship to response time and throughput. We outline the main components of an intelligent file system that performs striping on a file-specific basis by taking into account the requirements of the applications and performs load balancing by judicious file allocation and dynamic redistributions of the data when access patterns change. Our system uses simple but effective heuristics that incur only little overhead. We report on experiments based on real-life traces