DocumentCode :
598616
Title :
Combining in-situ and in-transit processing to enable extreme-scale scientific analysis
Author :
Bennett, Janine C. ; Abbasi, Hasan ; Bremer, Peer-Timo ; Grout, Ray ; Gyulassy, Attila ; Tong Jin ; Klasky, Scott ; Kolla, Hemanth ; Parashar, Manish ; Pascucci, V. ; Pebay, P. ; Thompson, Daniel ; Hongfeng Yu ; Fan Zhang ; Chen, Jiann-Jong
fYear :
2012
fDate :
10-16 Nov. 2012
Firstpage :
1
Lastpage :
9
Abstract :
With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations to a set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution of various analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. We demonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.
Keywords :
concurrency control; data visualisation; digital simulation; input-output programs; processor scheduling; resource allocation; statistical analysis; topology; ADIOS framework; DataSpaces framework; I/O cost mitigation constraints; Jaguar XK6; S3D; asynchronous data transfers; concurrent approach; data analysis pipeline; descriptive statistics; extreme-scale computing; extreme-scale scientific analysis; in-situ processing; in-transit processing; massively-parallel turbulent combustion code; offloading computations; persistent storage; primary compute resources; raw simulation data; resource scheduling system; secondary resources; temporal resolutions; topological analysis; visualization; Algorithm design and analysis; Analytical models; Computational modeling; Data models; Data transfer; Data visualization; Processor scheduling;
fLanguage :
English
Publisher :
ieee
Conference_Titel :
High Performance Computing, Networking, Storage and Analysis (SC), 2012 International Conference for
Conference_Location :
Salt Lake City, UT
ISSN :
2167-4329
Print_ISBN :
978-1-4673-0805-2
Type :
conf
DOI :
10.1109/SC.2012.31
Filename :
6468528
Link To Document :
بازگشت