Article

MPI-IO/GPFS, an optimized implementation of MPI-IO on top of GPFS

Authors:

Jean-Pierre Prost,

Richard Treumann,

Richard Hedges,

Bin Jia,

Alice KonigesAuthors Info & Claims

SC '01: Proceedings of the 2001 ACM/IEEE conference on Supercomputing

Page 17

https://doi.org/10.1145/582034.582051

Published: 10 November 2001 Publication History

Get Access

Abstract

MPI-IO/GPFS is an optimized prototype implementation of the I/O chapter of the Message Passing Interface (MPI) 2 standard. It uses the IBM General Parallel File System (GPFS) Release 3 as the underlying file system. This paper describes optimization features of the prototype that take advantage of new GPFS programming interfaces. It also details how collective data access operations have been optimized by minimizing the number of messages exchanged in sparse accesses and by increasing the overlap of communication with file access. Experimental results show a performance gain. A study of the impact of varying the number of tasks running on the same node is also presented.

References

[1]

{Mpif-97} Message Passing Interface Forum, MPI-2: Extensions to the Message-Passing Interface, July 18, 1997. http://www.mpi-forum.org/docs/docs.html.]]

Google Scholar

[2]

{GPFS-00} IBM General Parallel File System for AIX: Installation and Administration Guide, IBM Document SA22-7278-03, July 2000. http://www.rs6000.ibm.com/resource/aix_resource/sp_books/gpfs/insta ll_admin/install_admin_v1r/gpfs1mst.html.]]

Google Scholar

[3]

{Pros-00} J.-P. Prost, R. Treumann, R. Hedges, A. Koniges, and A. White, Towards a High-Performance Implementation of MPI-IO on top of GPFS, Sixth International Euro-Par Conference, Springer-Verlag, Aug.-Sep. 2000, pp. 1253-1262.]]

Digital Library

Google Scholar

[4]

{Thal-99} R. Thakur, W. Gropp, and E. Lusk, Data Sieving and Collective I/O in ROMIO, Seventh Symposium on the Frontiers of Massively Parallel Computation, IEEE Computer Society Press, Feb. 1999, pp. 182-189.]]

Digital Library

Google Scholar

[5]

{Tha2-99} R. Thakur, W. Gropp, and E. Lusk, On Implementing MPI-IO Portably and with High Performance, Sixth Workshop on Input/Output in Parallel and Distributed Systems, May 1999, pp. 23-32.]]

Digital Library

Google Scholar

Cited By

View all

Bez JTang HXie BWilliams-Young DLatham RRoss ROral SByna S(2021)I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW)10.1109/PDSW54622.2021.00008(15-22)Online publication date: Nov-2021
https://doi.org/10.1109/PDSW54622.2021.00008
Congiu GGrawinkel MPadua FMorse JSuB TBrinkmann A(2017)MERCURY: A Transparent Guided I/O Framework for High Performance I/O Stacks2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)10.1109/PDP.2017.83(46-53)Online publication date: 2017
https://doi.org/10.1109/PDP.2017.83
Artigues ACugnasco CBecerra YCucchietti FHouzeaux GVazquez MTorres JAyguadé ELabarta J(2017)ParaView + Alya + D8tree: Integrating High Performance Computing and High Performance Data AnalyticsProcedia Computer Science10.1016/j.procs.2017.05.170108(465-474)Online publication date: 2017
https://doi.org/10.1016/j.procs.2017.05.170
Show More Cited By

Index Terms

MPI-IO/GPFS, an optimized implementation of MPI-IO on top of GPFS

Recommendations

MPI-IO/Gfarm: An Optimized Implementation of MPI-IO for the Gfarm File System
CCGRID '11: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

This paper proposes a design and implementation of an MPI-IO implementation of the Gfarm file system, called MPI-IO/Gfarm. The Gfarm file system is a global file system that federates the local storage of compute nodes among several clusters. It has a ...
Unifying HDFS and GPFS: Enabling Analytics on Software-Defined Storage
Middleware '16: Proceedings of the 17th International Middleware Conference

Distributed file systems built for Big Data Analytics and cluster file systems built for traditional applications have very different functionality requirements, resulting in separate storage silos. In enterprises, there is often the need to run ...
Cashing in on hints for better prefetching and caching in PVFS and MPI-IO
HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

In this work, we propose, implement and test a novel approach to the management of parallel I/O in high-performance computing. Our proposed approach is built upon three complementary ideas: (i) allowing users to place hints into the application code ...

Comments

Information & Contributors

Information

Published In

SC '01: Proceedings of the 2001 ACM/IEEE conference on Supercomputing

November 2001

756 pages

ISBN:158113293X

DOI:10.1145/582034

Conference Chair:
Charles Slocomb
Los Alamos National Laboratory

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 November 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SC '01

Sponsor:

SIGARCH
IEEE-CS

SC '01: International Conference for High Performance Computing, Networking, Storage and Analysis

November 10 - 16, 2001

Colorado, Denver

Acceptance Rates

SC '01 Paper Acceptance Rate 60 of 240 submissions, 25%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
377
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Bez JTang HXie BWilliams-Young DLatham RRoss ROral SByna S(2021)I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW)10.1109/PDSW54622.2021.00008(15-22)Online publication date: Nov-2021
https://doi.org/10.1109/PDSW54622.2021.00008
Congiu GGrawinkel MPadua FMorse JSuB TBrinkmann A(2017)MERCURY: A Transparent Guided I/O Framework for High Performance I/O Stacks2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP)10.1109/PDP.2017.83(46-53)Online publication date: 2017
https://doi.org/10.1109/PDP.2017.83
Artigues ACugnasco CBecerra YCucchietti FHouzeaux GVazquez MTorres JAyguadé ELabarta J(2017)ParaView + Alya + D8tree: Integrating High Performance Computing and High Performance Data AnalyticsProcedia Computer Science10.1016/j.procs.2017.05.170108(465-474)Online publication date: 2017
https://doi.org/10.1016/j.procs.2017.05.170
Dorier MAntoniu GCappello FSnir MSisneros RYildiz OIbrahim SPeterka TOrf L(2016)DamarisACM Transactions on Parallel Computing10.1145/29873713:3(1-43)Online publication date: 25-Oct-2016
https://dl.acm.org/doi/10.1145/2987371
Yin YJin HSun X(2015)I/O and File Systems for Data-Intensive ApplicationsHandbook on Data Centers10.1007/978-1-4939-2092-1_18(561-582)Online publication date: 17-Mar-2015
https://doi.org/10.1007/978-1-4939-2092-1_18
Huang XWang WFu HYang GWang BZhang C(2014)A fast input/output library for high-resolution climate modelsGeoscientific Model Development10.5194/gmd-7-93-20147:1(93-103)Online publication date: 14-Jan-2014
https://doi.org/10.5194/gmd-7-93-2014
Dorier MAntoniu GRoss RKimpe DIbrahim S(2014)CALCioMProceedings of the 2014 IEEE 28th International Parallel and Distributed Processing Symposium10.1109/IPDPS.2014.27(155-164)Online publication date: 19-May-2014
https://dl.acm.org/doi/10.1109/IPDPS.2014.27
Li XXiao LQiu MDong BRuan L(2014)Enabling dynamic file I/O path selection at runtime for parallel file systemThe Journal of Supercomputing10.1007/s11227-013-1077-668:2(996-1021)Online publication date: 1-May-2014
https://dl.acm.org/doi/10.1007/s11227-013-1077-6
Tsujita YYoshinaga KHori ASato MNamiki MIshikawa Y(2014)Improving Parallel I/O Performance Using Multithreaded Two-Phase I/O with Processor Affinity ManagementParallel Processing and Applied Mathematics10.1007/978-3-642-55224-3_67(714-723)Online publication date: 6-May-2014
https://doi.org/10.1007/978-3-642-55224-3_67
Huang XWang WFu HYang GWang BZhang C(2013)A fast input/output library for high resolution climate modelsGeoscientific Model Development Discussions10.5194/gmdd-6-4775-20136:3(4775-4807)Online publication date: 13-Sep-2013
https://doi.org/10.5194/gmdd-6-4775-2013
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Cited By

Index Terms

Recommendations

MPI-IO/Gfarm: An Optimized Implementation of MPI-IO for the Gfarm File System

Unifying HDFS and GPFS: Enabling Analytics on Software-Defined Storage

Cashing in on hints for better prefetching and caching in PVFS and MPI-IO

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

MPI-IO/Gfarm: An Optimized Implementation of MPI-IO for the Gfarm File System

Unifying HDFS and GPFS: Enabling Analytics on Software-Defined Storage

Cashing in on hints for better prefetching and caching in PVFS and MPI-IO

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations