Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context
Authors: 
Ranjan Sarpangala Venkatesh, 
Greg Eisenhauer, Scott Klasky, Ada GavrilovskaAuthors Info & Claims
Ranjan Sarpangala Venkatesh, Greg Eisenhauer, Scott Klasky, Ada GavrilovskaAuthors Info & ClaimsPages 1223 - 1228
Abstract
In HPC, data movement between applications is typically facilitated by I/O middleware, such as the Adaptable I/O System (ADIOS). This middleware leverages the capabilities of the underlying storage services, to facilitate data movement and distribution. A recent storage system, Intel DAOS, promises to deliver new capabilities for achieving performance and scalability on emerging memory/storage systems. DAOS has already deployed in University of Cambridge, TACC, and the upcoming Aurora supercomputer. This paper investigates the performance tradeoffs associated with mapping ADIOS over one of the many different DAOS interfaces and data models, and makes recommendations for their efficient use.
References
[1]
Sadaf R Alam, Hussein N El-Harake, Kristopher Howard, Neil Stringfellow, and Fabio Verzelloni. 2011. Parallel I/O and the metadata wall. In Proceedings of the sixth workshop on Parallel Data Storage. 13–18.
[2]
Michael Hennecke. 2020. Daos: A scale-out high performance storage stack for storage class memory. Supercomputing frontiers (2020), 40.
[3]
Jialin Liu, Quincey Koziol, Gregory F. Butler, Neil Fortner, Mohamad Chaarawi, Houjun Tang, Suren Byna, Glenn K. Lockwood, Ravi Cheema, Kristy A. Kallback-Rose, Damian Hazen, and Mr Prabhat. 2018. Evaluation of HPC Application I/O on Object Storage Systems. In 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS). 24–34. https://doi.org/10.1109/PDSW-DISCS.2018.00005
[4]
Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, 2014. Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience 26, 7 (2014), 1453–1473.
[5]
Ricardo Macedo, Mariana Miranda, Yusuke Tanimura, Jason Haga, Amit Ruhela, Stephen Lien Harrell, Richard Todd Evans, José Pereira, and Joao Paulo. 2023. Taming Metadata-intensive HPC Jobs Through Dynamic, Application-agnostic QoS Control. In 23nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE.
[6]
Jerome Soumagne, Jordan Henderson, Mohamad Chaarawi, Neil Fortner, Scot Breitenfeld, Songyu Lu, Dana Robinson, Elena Pourmal, and Johann Lombardi. 2021. Accelerating hdf5 i/o for exascale using daos. IEEE Transactions on Parallel and Distributed Systems 33, 4 (2021), 903–914.
[7]
Ranjan Sarpangala Venkatesh, Tony Mason, Pradeep Fernando, Greg Eisenhauer, and Ada Gavrilovska. 2021. Scheduling HPC Workflows with Intel Optane Persistent Memory. In 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 56–65.
Index Terms
- Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context
Recommendations
Data management services and transfer scheme in ChinaGrid
ChinaGrid is a nationwide grid project supported by the Chinese government that aims at integrating massive heterogeneous resources distributed on the China Education and Research Network (CERNET). The ChinaGrid Supporting Platform (CGSP) is a grid ...
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks
Applications running on leadership platforms are more and more bottlenecked by storage input/output I/O. In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System ADIOS in 2005. ...
UNICORE in XSEDE: Through Development, Integration, Deployment and beyond
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery EnvironmentIn this paper we discuss the implementation of UNICORE in XSEDE. UNICORE is a Grid middleware tool that was identified by XSEDE to further the areas of remote job submission, campus bridging and workflows. We talk about the overall architecture of ...
Comments
Information & Contributors
Information
Published In
November 2023
2180 pages
ISBN:9798400707858
DOI:10.1145/3624062
Copyright © 2023 ACM.
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SC-W 2023
SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis
November 12 - 17, 2023
CO, Denver, USA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 145Total Downloads
- Downloads (Last 12 months)145
- Downloads (Last 6 weeks)33
Reflects downloads up to 01 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in