Hierarchical scheduling of independent tasks with shared files

Parallel computing platforms such as grids, clusters and multi-clusters constitute promising alternatives for executing applications comprised by a large number of independent tasks. However, some application and architectural characteristics may severely limit performance gains. For instance, tasks with fine granularity, huge data files to be transmitted to or from data repositories, and tasks which share common input files are examples of such characteristics that may cause poor performance. Bottlenecks may also appear due to the existence of a centralized controller in the master-slave architecture, or centralized data repositories within the system. This paper shows how system efficiency decreases under such conditions. To overcome such limitations, a hierarchical strategy for file distribution which aims at improving the system capacity of delivering input files to processing nodes is proposed and assessed. Such a strategy arranges the processors in a tree topology, clusters tasks that share common input files together, and maps such groups of tasks to clusters of processors. By means of such strategy, significant improvements in the application scalability can be achieved