Collaborative monitoring and visualization of HPC data

As simulations begin to scale to extreme processor counts trying to understand the mysteries of the universe, collaboration becomes an essential piece of the scientists´ daily life as they work to run, analyze, and process their data from these simulations. Most of the teams that we collaborate with work identically to the way they did in the past, without using effective collaborative tools to share their knowledge with their peers. Collaboration is usually an afterthought, and is often handled in an awkward setting. We believe the best way to introduce collaboration into areas that are resistant by nature is to embed into low level and hidden system components so that scientists collaborate without consciously putting in extra efforts. Based on this hypothesis, this paper presents our work in creating a collaborative system, which allows a diverse set of scientists to work together efficiently. Two of the main aspects of our system are in our use of provenance to associate files and its associated metadata with the information that domain experts are interested in, and an easy-to-use high-performance I/O system which automatically annotates the output file(s) with a unified schema. To accomplish our goals, we leverage an existing I/O framework, ADIOS and an existing web interface, eSiMon, and add new techniques and mechanism to efficiently bring together computation and visualization.