short-paper

The challenges of elastic in situ analysis and visualization

Authors:

Matthieu Dorier,

Robert RossAuthors Info & Claims

ISAV '19: Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

Pages 23 - 28

https://doi.org/10.1145/3364228.3364234

Published: 18 November 2019 Publication History

Abstract

In situ analysis and visualization have been proposed in high-performance computing (HPC) to enable executing analysis tasks while a simulation is running, bypassing the parallel file system and avoiding the need for storing massive amounts of data. One aspect of in situ analysis that has not been extensively researched to date, however, is elasticity. Current in situ analysis frameworks use a fixed amount of resources and can hardly be scaled up or down dynamically throughout the simulation's run time as a response to changes in the requirements.

In this paper, we present the challenges posed by elastic in situ analysis and visualization. We emphasize that elasticity can take various forms. We show the difficulties of supporting each form of elasticity with the state-of-the-art HPC technologies, and we suggest solutions to overcome these difficulties. The resulting four-way classification can be seen as a taxonomy for future elastic in situ analysis and visualization systems.

References

[1]

2019. ASCR Workshop on In Situ Data Management. https://science.osti.gov/-/media/ascr/pdf/programdocuments/docs/2019/ISDM_brochure.PDF.

[2]

Utkarsh Ayachit, Andrew Bauer, Berk Geveci, Patrick O'Leary, Kenneth Moreland, Nathan Fabian, and Jeffrey Mauldin. 2015. ParaView Catalyst: Enabling in situ data analysis and visualization. In Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization. ACM, 25--29.

Digital Library

[3]

Utkarsh Ayachit, Brad Whitlock, Matthew Wolf, Burlen Loring, Berk Geveci, David Lonie, and E Bethel. 2016. The SENSEI generic in situ interface. In Proceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization. IEEE Press, 40--44.

Digital Library

[4]

Ilya Baldin, Jeff Chase, Yufeng Xin, Anirban Mandal, Paul Ruth, Claris Castillo, Victor Orlikowski, Chris Heermann, and Jonathan Mills. 2016. ExoGENI: A multi-domain infrastructure-as-a-service testbed. In The GENI Book. Springer, 279--315.

[5]

David A Boyuka, Sriram Lakshminarasimham, Xiaocheng Zou, Zhenhuan Gong, John Jenkins, Eric R Schendel, Norbert Podhorszki, Qing Liu, Scott Klasky, and Nagiza F Samatova. 2014. Transparent in situ data transformations in adios. In 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. IEEE, 256--266.

Digital Library

[6]

Ralph H Castain, Joshua Hursey, Aurelien Bouteiller, and David Solt. 2018. PMIX: Process management for exascale environments. Parallel Comput. 79 (2018), 9--29.

[7]

Leonardo Dagum and Ramesh Menon. 1998. OpenMP: An industry-standard API for shared-memory programming. Computing in Science & Engineering 1 (1998), 46--55.

Digital Library

[8]

Jai Dayal, Drew Bratcher, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Xuechen Zhang, Hasan Abbasi, Scott Klasky, and Norbert Podhorszki. 2014. Flexpath: Type-based publish/subscribe system for large-scale science analytics. In 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. IEEE, 246--255.

Digital Library

[9]

Estelle Dirand, Laurent Colombet, and Bruno Raffin. 2018. TINS: A task-based dynamic helper core strategy for in situ analytics. In Asian Conference on Super-computing Frontiers. Springer, 159--178.

[10]

Matthieu Dorier, Gabriel Antoniu, Franck Cappello, Marc Snir, Robert Sisneros, Orcun Yildiz, Shadi Ibrahim, Tom Peterka, and Leigh Orf. 2016. Damaris: Addressing performance variability in data management for post-petascale simulations. ACM Transactions on Parallel Computing (TOPC) 3, 3 (2016), 15.

[11]

Matthieu Dorier, Philip Carns, Kevin Harms, Robert Latham, Robert Ross, Shane Snyder, Justin Wozniak, Samuel Gutierrez, Bob Robey, Brad Settlemyer, Galen Shipman, Jerome Soumagne, James Kowalkowski, Marc Paterno, and Saba Sehrish. 2018. Methodology for the rapid development of scalable HPC data services. In Proceedings of the PDSW-DISC 2018 workshop (SC18). https://sc18.supercomputing.org/proceedings/workshops/workshop_pages/ws_pdsw106.html

[12]

Matthieu Dorier, Roberto R. Sisneros, Tom Peterka, Gabriel Antoniu, and Dave B. Semeraro. 2013. Damaris/Viz: a nonintrusive, adaptable and user-friendly in situ visualization framework. In IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV). Atlanta, United States. https://hal.inria.fr/hal-00859603

[13]

Matthieu Dorier, Justin M Wozniak, and Robert Ross. 2017. Supporting task-level fault-tolerance in HPC workflows by launching MPI jobs inside MPI jobs. In Proceedings of the 12th Workshop on Workflows in Support of Large-Scale Science. ACM, 5.

Digital Library

[14]

Matthieu Dreher and Tom Peterka. 2016. Bredala: Semantic data redistribution for in situ applications. In 2016 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 279--288.

[15]

Matthieu Dreher and Tom Peterka. 2017. Decaf: Decoupled dataflows for in situ high-performance workflows. Technical Report. Argonne National Laboratory, Argonne, IL (United States).

[16]

OpenFabrics Working Group et al. [n. d.]. Libfabric. https://www.openfabrics.org/.

[17]

Abhishek Gupta, Bilge Acun, Osman Sarood, and Laxmikant V Kalé. 2014. Towards realizing the potential of malleable jobs. In 2014 21st International Conference on High Performance Computing (HiPC). IEEE, 1--10.

[18]

Daniel Holmes, Kathryn Mohror, Ryan E. Grant, Anthony Skjellum, Martin Schulz, Wesley Bland, and Jeffrey M. Squyres. 2016. MPI sessions: leveraging runtime infrastructure to increase scalability of applications at exascale. In Proceedings of the 23rd European MPI Users' Group Meeting (EuroMPI 2016). ACM, New York, NY, USA, 121--129.

Digital Library

[19]

Atsushi Hori, Min Si, Balazs Gerofi, Masamichi Takagi, Jai Dayal, Pavan Balaji, and Yutaka Ishikawa. 2018. Process-in-process: techniques for practical address-space sharing. In Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing. ACM, 131--143.

Digital Library

[20]

Anubhav Jain, Shyue Ping Ong, Wei Chen, Bharat Medasani, Xiaohui Qu, Michael Kocher, Miriam Brafman, Guido Petretto, Gian-Marco Rignanese, Geoffroy Hautier, et al. 2015. FireWorks: A dynamic workflow system designed for high-throughput applications. Concurrency and Computation: Practice and Experience 27, 17 (2015), 5037--5059.

Digital Library

[21]

T Kuhlen, R Pajarola, and K Zhou. 2011. Parallel in situ coupling of simulation with a fully featured visualization system. In Proceedings of the 11th Eurographics Conference on Parallel Graphics and Visualization (EGPGV).

Digital Library

[22]

Erich Lohrmann, Zarija Lukić, Dmitriy Morozov, and Juliane Müller. 2017. Programmable in situ system for iterative workflows. In Workshop on Job Scheduling Strategies for Parallel Processing. Springer, 122--131.

[23]

K. Moreland, C. Sewell, W. Usher, L. Lo, J. Meredith, D. Pugmire, J. Kress, H. Schroots, K. Ma, H. Childs, M. Larsen, C. Chen, R. Maynard, and B. Geveci. 2016. VTK-m: Accelerating the visualization toolkit for massively threaded architectures. IEEE Computer Graphics and Applications 36, 3 (May 2016), 48--58.

Digital Library

[24]

Dmitriy Morozov and Zarija Lukic. 2016. Master of puppets: Cooperative multitasking for in situ processing. In Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing. ACM, 285--288.

Digital Library

[25]

Bradford Nichols, Dick Buttlar, Jacqueline Farrell, and Jackie Farrell. 1996. Pthreads programming: A POSIX standard for better multiprocessing. "O'Reilly Media, Inc.".

[26]

M Parashar, PT Bremer, K Heitmann, M Larsen, J Patchett, T Peterka, M Srinivasan, N Röber, S Frey, and B Raffin. 2019. 4.2 Workflow Specification. In Situ Visualization for Computational Science (2019), 13.

[27]

J Patchett, H Childs, A Bauer, PT Bremer, T Carrard, M Dorier, K Heitmann Garth, K Moreland, T Peterka, D Pleiter, et al. 2019. 4.3 Workflow Execution. In Situ Visualization for Computational Science (2019), 16.

[28]

Tom Peterka, Robert Ross, Attila Gyulassy, Valerio Pascucci, Wesley Kendall, Han-Wei Shen, Teng-Yok Lee, and Abon Chaudhuri. 2011. Scalable parallel building blocks for custom data analysis. In 2011 IEEE Symposium on Large Data Analysis and Visualization. IEEE, 105--112.

[29]

Paul Ruth, Anirban Mandal, Yufeng Xin, Ilia Baldine, Chris Heerman, and Jeff Chase. 2012. Dynamic network provisioning for data intensive applications in the cloud. In 2012 IEEE 8th International Conference on E-Science. IEEE, 1--2.

Digital Library

[30]

Sangmin Seo, Abdelhalim Amer, Pavan Balaji, Cyril Bordage, George Bosilca, Alex Brooks, Philip Carns, Adrián Castelló, Damien Genet, Thomas Herault, et al. 2017. Argobots: A lightweight low-level threading and tasking framework. IEEE Transactions on Parallel and Distributed Systems 29, 3 (2017), 512--526.

[31]

Jerome Soumagne, Dries Kimpe, Judicael Zounmevo, Mohamad Chaarawi, Quincey Koziol, Ahmad Afsahi, and Robert Ross. 2013. Mercury: Enabling remote procedure call for high-performance computing. In 2013 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 1--8.

[32]

Enric Tejedor, Yolanda Becerra, Guillem Alomar, Anna Queralt, Rosa M Badia, Jordi Torres, Toni Cortes, and Jesús Labarta. 2017. PyCOMPSs: Parallel computational workflows in Python. The International Journal of High Performance Computing Applications 31, 1 (2017), 66--82.

Digital Library

[33]

Théophile Terraz, Alejandro Ribes, Yvan Fournier, Bertrand Iooss, and Bruno Raffin. 2017. Melissa: Large scale in transit sensitivity analysis avoiding intermediate files. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 61.

Digital Library

[34]

Wil MP Van Der Aalst and Arthur HM Ter Hofstede. 2005. YAWL: Yet another workflow language. Information systems 30, 4 (2005), 245--275.

[35]

Thomas Willhalm and Nicolae Popovici. 2008. Putting intel® threading building blocks to work. In Proceedings of the 1st international workshop on Multicore software engineering. ACM, 3--4.

Digital Library

[36]

Justin M Wozniak, Timothy G Armstrong, Michael Wilde, Daniel S Katz, Ewing Lusk, and Ian T Foster. 2013. Swift/t: Large-scale application composition via distributed-memory dataflow processing. In 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing. IEEE, 95--102.

Digital Library

[37]

Orcun Yildiz, Jorge Ejarque, Henry Chan, Subramanian Sankaranarayanan, Rosa M Badia, and Tom Peterka. 2019. Heterogeneous hierarchical workflow composition. Computing in Science & Engineering (2019).

[38]

Qing Zheng, Charles D Cranor, Danhao Guo, Gregory R Ganger, George Amvrosiadis, Garth A Gibson, Bradley W Settlemyer, Gary Grider, and Fan Guo. 2018. Scaling embedded in-situ indexing with deltaFS. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 30--44.

Digital Library

Cited By

Yildiz OGueroudji ABigot JRaffin BBadia RPeterka T(2024)Extreme-scale workflows: A perspective from the JLESC international communityFuture Generation Computer Systems10.1016/j.future.2024.07.041161(502-513)Online publication date: Dec-2024
https://doi.org/10.1016/j.future.2024.07.041
Ju YHuber DPerez AUlbl PMarkidis SSchlatter PSchulz MSchreiber MLaure E(2024)Dynamic Resource Management for In-Situ Techniques Using MPI-SessionsRecent Advances in the Message Passing Interface10.1007/978-3-031-73370-3_7(105-120)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73370-3_7
Bruder VLarsen MErtl TChilds HFrey S(2023)A Hybrid in Situ Approach for Cost Efficient Image Database GenerationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.316959029:9(3788-3798)Online publication date: 1-Sep-2023
https://doi.org/10.1109/TVCG.2022.3169590
Show More Cited By

Index Terms

The challenges of elastic in situ analysis and visualization
1. Computing methodologies
  1. Distributed computing methodologies
    1. Distributed algorithms
  2. Parallel computing methodologies
    1. Parallel algorithms
      1. Self-organization

Recommendations

Interactive in Situ Visualization
High Performance Computing. ISC High Performance 2024 International Workshops
Abstract
To achieve interactive in situ visualization in immersive virtual environments, the parallel visualization tool Vistle has been coupled with the SENSEI in situ framework. Vistle’s modular architecture, designed for distributed systems, allows in ...
Catalyst Revised: Rethinking the ParaView in Situ Analysis and Visualization API
High Performance Computing
Abstract
As in situ analysis goes mainstream, ease of development, deployment, and maintenance becomes essential, perhaps more so than raw capabilities. In this paper, we present the design and implementation of Catalyst, an API for in situ analysis using ...
In-Situ Visualization with Membrane Layer for Movie-Based Visualization
Computational Science – ICCS 2019
Abstract
We propose a movie-based visualization for High Performance Computing (HPC) visualization. In this method, a viewer interactively explores a movie database with a specially designed application program called a movie data browser. The database is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ISAV '19: Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

November 2019

56 pages

ISBN:9781450377232

DOI:10.1145/3364228

General Chairs:
Ken Moreland
Sandia National Laboratory
,
Christoph Garth
Technische Universität Kaiserslautern, Germany
,
Program Chairs:
E. Wes Bethel
Lawrence Berkeley National Laboratory
,
Bruno Raffin
INRIA, France

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ISAV'19

ISAV'19: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

November 18, 2019

Colorado, Denver, USA

Acceptance Rates

Overall Acceptance Rate 23 of 63 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
232
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)2

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yildiz OGueroudji ABigot JRaffin BBadia RPeterka T(2024)Extreme-scale workflows: A perspective from the JLESC international communityFuture Generation Computer Systems10.1016/j.future.2024.07.041161(502-513)Online publication date: Dec-2024
https://doi.org/10.1016/j.future.2024.07.041
Ju YHuber DPerez AUlbl PMarkidis SSchlatter PSchulz MSchreiber MLaure E(2024)Dynamic Resource Management for In-Situ Techniques Using MPI-SessionsRecent Advances in the Message Passing Interface10.1007/978-3-031-73370-3_7(105-120)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73370-3_7
Bruder VLarsen MErtl TChilds HFrey S(2023)A Hybrid in Situ Approach for Cost Efficient Image Database GenerationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.316959029:9(3788-3798)Online publication date: 1-Sep-2023
https://doi.org/10.1109/TVCG.2022.3169590
Dorier MWang ZRamesh SAyachit USnyder SRoss RParashar M(2023)Towards elastic in situ analysis for high-performance computing simulationsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.02.014177:C(106-116)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.jpdc.2023.02.014
Pugmire DHuang JMoreland KKlasky S(2023)The Need for Pervasive In Situ Analysis and Visualization (P-ISAV)High Performance Computing. ISC High Performance 2022 International Workshops10.1007/978-3-031-23220-6_21(306-316)Online publication date: 4-Jan-2023
https://doi.org/10.1007/978-3-031-23220-6_21
Wang ZDorier MParashar M(2022)Research Perspectives Toward Autonomic Optimization of In Situ Analysis and Visualization2022 IEEE/ACM International Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV)10.1109/ISAV56555.2022.00007(7-13)Online publication date: Nov-2022
https://doi.org/10.1109/ISAV56555.2022.00007
Dorier MWang ZAyachit USnyder SRoss RParashar M(2022)Colza: Enabling Elastic In Situ Visualization for High-performance Computing Simulations2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS53621.2022.00059(538-548)Online publication date: May-2022
https://doi.org/10.1109/IPDPS53621.2022.00059
Yildiz ODreher MPeterka T(2022)Decaf: Decoupled Dataflows for In Situ WorkflowsIn Situ Visualization for Computational Science10.1007/978-3-030-81627-8_7(137-158)Online publication date: 5-May-2022
https://doi.org/10.1007/978-3-030-81627-8_7
Usher WPark HLee MNavrátil PFussell DPascucci V(2022)A Simulation-Oblivious Data Transport Model for Flexible In Transit VisualizationIn Situ Visualization for Computational Science10.1007/978-3-030-81627-8_18(399-419)Online publication date: 5-May-2022
https://doi.org/10.1007/978-3-030-81627-8_18
Singhal SSussman AWolf MMehta KChoi J(2021)DYFLOW: A flexible framework for orchestrating scientific workflows on supercomputers50th International Conference on Parallel Processing Workshop10.1145/3458744.3474037(1-11)Online publication date: 9-Aug-2021
https://dl.acm.org/doi/10.1145/3458744.3474037
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten