Title :
Nimrod/K: Towards massively parallel dynamic Grid workflows
Author :
Abramson, David ; Enticott, Colin ; Altinas, Ilkay
Author_Institution :
Fac. of Inf. Technol., Monash Univ., Clayton, VIC, Australia
Abstract :
A challenge for Grid computing is the difficulty in developing software that is parallel, distributed and highly dynamic. Whilst there have been many general purpose mechanisms developed over the years, Grid programming still remains a low level, error prone task. Scientific workflow engines can double as programming environments, and allow a user to compose dasiavirtualpsila Grid applications from pre-existing components. Whilst existing workflow engines can specify arbitrary parallel programs, (where components use message passing) they are typically not effective with large and variable parallelism. Here we discuss dynamic dataflow, originally developed for parallel tagged dataflow architectures (TDAs), and show that these can be used for implementing Grid workflows. TDAs spawn parallel threads dynamically without additional programming. We have added TDAs to Kepler, and show that the system can orchestrate workflows that have large amounts of variable parallelism. We demonstrate the system using case studies in chemistry and in cardiac modelling.
Keywords :
grid computing; message passing; multi-threading; natural sciences computing; TDA; distributed software; message passing; parallel dynamic grid computing; parallel program; parallel software; parallel thread; scientific workflow engine; tagged dataflow architecture; Application software; Chemistry; Dynamic programming; Engines; Grid computing; Message passing; Parallel processing; Parallel programming; Programming environments; Yarn;
Conference_Titel :
High Performance Computing, Networking, Storage and Analysis, 2008. SC 2008. International Conference for
Conference_Location :
Austin, TX
Print_ISBN :
978-1-4244-2834-2
Electronic_ISBN :
978-1-4244-2835-9
DOI :
10.1109/SC.2008.5215726