Article

Large files, small writes, and pNFS

Authors:

Dean Hildebrand,

Peter HoneymanAuthors Info & Claims

ICS '06: Proceedings of the 20th annual international conference on Supercomputing

Pages 116 - 124

https://doi.org/10.1145/1183401.1183419

Published: 28 June 2006 Publication History

Abstract

Workload characterization studies highlight the prevalence of small and sequential data requests in scientific applications. Parallel file systems excel at large data transfers but sometimes at the expense of small I/O performance. pNFS is an NFSv4.1 high-performance enhancement that provides direct storage access to parallel file systems while preserving NFSv4 operating system and hardware platform independence. This paper demonstrates that distributed file systems can increase write throughput to parallel data stores---regardless of file size---by overcoming parallel file system inefficiencies. We also show how pNFS can improve the overall write performance of parallel file systems by using direct, parallel I/O for large write requests and a distributed file system for small write requests. We describe our pNFS prototype and present experiments demonstrating the performance improvements.

References

[1]

Cluster File Systems Inc., "Lustre: A Scalable, High-Performance File System," 2002.]]

[2]

PVFS2 Development Team, "Parallel Virtual File System, Version 2," www.pvfs.org/pvfs2.]]

[3]

F. Schmuck and R. Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters," in Proceedings of the USENIX Conference on File and Storage Technologies, 2002.]]

Digital Library

[4]

Panasas Inc., "Panasas ActiveScale File System Datasheet," www.panasas.com, 2003.]]

[5]

D. Kotz and N. Nieuwejaar, "Dynamic File-Access Characteristics of a Production Parallel Scientific Workload," in Proceedings of Supercomputing '94, 1994.]]

Digital Library

[6]

A. Purakayastha, C. Schlatter Ellis, D. Kotz, N. Nieuwejaar, and M. Best, "Characterizing Parallel File-Access Patterns on a Large-Scale Multiprocessor," in Proceedings of the Ninth International Parallel Processing Symposium, 1995.]]

Digital Library

[7]

N. Nieuwejaar, D. Kotz, A. Purakayastha, C. Schlatter Ellis, and M. Best, "File-Access Characteristics of Parallel Scientific Workloads," IEEE Transactions on Parallel and Distributed Systems, (7)10, pp. 1075--1089, 1996.]]

Digital Library

[8]

E. Smirni and D. A. Reed, "Workload Characterization of Input/Output Intensive Parallel Applications," in Proceedings of the Conference on Modeling Techniques and Tools for Computer Performance Evaluation, 1997.]]

Digital Library

[9]

E. Smirni, R. A. Aydt, A. A. Chien, and D. A. Reed, "I/O Requirements of Scientific Applications: An Evolutionary View," in Proceedings of the Fifth IEEE Conference on High Performance Distributed Computing, 1996.]]

Digital Library

[10]

P. E. Crandall, R. A. Aydt, A. A. Chien, and D. A. Reed, "Input/Output Characteristics of Scalable Parallel Applications," in Proceedings of Supercomputing '95, 1995.]]

Digital Library

[11]

F. Wang, Q. Xin, B. Hong, S. A. Brandt, E. L. Miller, and D. D. E Long, "File System Workload Analysis For Large Scale Scientific Computing Applications," in Proceedings of the 21st IEEE/12th NASA Goddard Conference on Mass Storage Systems and Technologies, 2004.]]

[12]

ASCI Purple RFP, www.llnl.gov/asci/platforms/purple/rfp.]]

[13]

R. Rew and G. Davis, "The Unidata netCDF: Software for Scientific Data Access," in Proceedings of the Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography and Hydrology, Anaheim, CA, 1990.]]

[14]

NCSA, "HDF5", hdf.ncsa.uiuc.edu/HDF5.]]

[15]

Sun Microsystems Inc., "NFS: Network File System Protocol Specification," RFC 1094, 1989.]]

Digital Library

[16]

Common Internet File System File Access Protocol, msdn.microsoft.com/library/en-us/cifs/protocol/cifs.asp.]]

[17]

M. G. Baker, J. H. Hartman, M. D. Kupfer, K. W. Shirriff, and J. K. Ousterhout, "Measurements of a Distributed File System," in Proceedings of the Thirteenth Symposium on Operating Systems Principles, 1991.]]

Digital Library

[18]

S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, and D. Noveck, "Network File System Version 4 Protocol Specification," RFC 3530, 2003.]]

Digital Library

[19]

B. Welch, B. Halevy, D. Black, A. Adamson, and D. Noveck, "pNFS Operations Summary," Internet Draft, draft-welch-pnfs-ops-00.txt, 2004.]]

[20]

G. Gibson, B. Welch, G. Goodson, and P. Corbett, "Parallel NFS Requirements and Design Considerations," Internet Draft, draft-gibson-pnfs-reqs-00.txt, 2004.]]

[21]

D. Hildebrand and P. Honeyman, "Exporting Storage Systems in a Scalable Manner with pNFS," in Proceedings of the 22nd IEEE - 13th NASA Goddard Conference on Mass Storage Systems and Technologies, Monterey, CA, 2005.]]

Digital Library

[22]

M. Rosenblum and J. K. Ousterhout, "The Design and Implementation of a Log-Structured File System," ACM Transactions on Computer Systems, (10)1, pp. 26--52, 1992.]]

Digital Library

[23]

J. H. Hartman and J. K. Ousterhout, "The Zebra Striped Network File System," ACM Transactions on Computer Systems, (13)3, 1995.]]

Digital Library

[24]

P. F. Corbett and D. G. Feitelson, "The Vesta Parallel File System," ACM Transactions on Computer Systems, (14)3, pp. 225--264, 1996.]]

Digital Library

[25]

EMC Celerra HighRoad Whitepaper, www.emc.com, 2001.]]

[26]

A. L. Drapeau, K. Shirriff, E. K. Lee, J. H. Hartman, E. L. Miller, S. Seshan, R. H. Katz, K. Lutz, D. A. Patterson, P. H. Chen, and G. A. Gibson, "RAID-II: A High-Bandwidth Network File Server," in Proceedings of the 21st International Symposium on Computer Architecture, 1994.]]

Digital Library

[27]

W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitzberg, W. Saphir, and M. Snir, MPI: The Complete Reference, volume 2--The MPI-2 Extensions. Cambridge, MA, 1998.]]

[28]

R. Thakur, W. Gropp, and E. Lusk, "Data Sieving and Collective I/O in ROMIO," in Proceedings of the 7th Symposium on the Frontiers of Massively Parallel Computation, 1999.]]

Digital Library

[29]

J. Satran, D. Smith, K. Meth, O. Biran, J. Hafner, C. Sapuntzakis, M. Bakke, M. Wakeley, L. Dalle Ore, P. Von Stamwitz, R. Haagens, M. Chadalapaka, E. Zeidner, and Y. Klein, "iSCSI," Internet Draft, draft-ietf-ips-iscsi-08.txt, 2001.]]

[30]

R. Brightwell, A. B. Maccabe, R. Riesen, and T. Hudson, "The Portals 3.3 Message Passing Interface," 2003.]]

[31]

R. Srinivasan, "RPC: Remote Procedure Call Protocol Specification Version 2," RFC 1831, 1995.]]

Digital Library

[32]

Unidata Program Center, "Where is NetCDF Used?," www.unidata.ucar.edu/software/netcdf/usage.html.]]

[33]

J. Katcher, "PostMark: A New File System Benchmark," Technical Report TR3022, Network Appliance, 1997.]]

[34]

IOR Benchmark, www.llnl.gov/asci/purple/benchmarks/limited/ior.]]

[35]

W. D. Norcott and D. Capps, "IOZone Filesystem Benchmark," 2003.]]

[36]

FLASH I/O Benchmark, flash.uchicago.edu/~jbgallag/io_bench.]]

[37]

ATLAS, atlasinfo.cern.ch.]]

[38]

The Large Hadron Collider, lhc.web.cern.ch.]]

Cited By

Wei QChen CXue MYang J(2015)Z-MAPACM Transactions on Storage10.1145/262966311:1(1-33)Online publication date: 24-Feb-2015
https://dl.acm.org/doi/10.1145/2629663
Wei QChen CYang J(2014)CBM: A cooperative buffer management for SSD2014 30th Symposium on Mass Storage Systems and Technologies (MSST)10.1109/MSST.2014.6855545(1-12)Online publication date: Jun-2014
https://doi.org/10.1109/MSST.2014.6855545
Hatzieleftheriou AAnastasiadis S(2013)Improving Bandwidth Efficiency for Consistent Multistream StorageACM Transactions on Storage10.1145/2435204.24352069:1(1-27)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.1145/2435204.2435206
Show More Cited By

Index Terms

Large files, small writes, and pNFS
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement

Recommendations

Direct-pNFS: scalable, transparent, and versatile access to parallel file systems
HPDC '07: Proceedings of the 16th international symposium on High performance distributed computing

Grid computations require global access to massive data stores. To meet this need, the GridNFS project aims to provide scalable, high-performance, transparent, and secure wide-area data management as well as a scalable and agile name space.

While ...
pNFS/PVFS2 over InfiniBand: early experiences
PDSW '07: Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing '07

The computing power of clusters has been rapidly growing up towards petascale capability, which requires petascale I/O systems to provide data in a sustained high-throughput manner. Network File System (NFS), a ubiquitous standard used in most existing ...
RFC 6688: Parallel NFS (pNFS) Block Disk Protection

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICS '06: Proceedings of the 20th annual international conference on Supercomputing

June 2006

385 pages

ISBN:1595932828

DOI:10.1145/1183401

General Chairs:
Greg Egan
Monash University
,
Yoichi Muraoka
Waseda University

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICS06

Sponsor:

ICS06: International Conference on Supercomputing 2006

June 28 - July 1, 2006

Queensland, Cairns, Australia

Acceptance Rates

ICS '06 Paper Acceptance Rate 37 of 141 submissions, 26%;

Overall Acceptance Rate 629 of 2,180 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
678
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei QChen CXue MYang J(2015)Z-MAPACM Transactions on Storage10.1145/262966311:1(1-33)Online publication date: 24-Feb-2015
https://dl.acm.org/doi/10.1145/2629663
Wei QChen CYang J(2014)CBM: A cooperative buffer management for SSD2014 30th Symposium on Mass Storage Systems and Technologies (MSST)10.1109/MSST.2014.6855545(1-12)Online publication date: Jun-2014
https://doi.org/10.1109/MSST.2014.6855545
Hatzieleftheriou AAnastasiadis S(2013)Improving Bandwidth Efficiency for Consistent Multistream StorageACM Transactions on Storage10.1145/2435204.24352069:1(1-27)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.1145/2435204.2435206
Santos EComunello Ede Macedo DCapretz MPrado TDantas M(2013)An analysis of replication and retrieval of medical image data using a database management system and a distributed file system2013 IEEE Symposium on Computers and Communications (ISCC)10.1109/ISCC.2013.6755049(000814-000819)Online publication date: Jul-2013
https://doi.org/10.1109/ISCC.2013.6755049
Wang BJiang JYang G(2013)tpNFSProceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing Workshops and PhD Forum10.1109/IPDPSW.2013.36(1989-1996)Online publication date: 20-May-2013
https://dl.acm.org/doi/10.1109/IPDPSW.2013.36
Bai XJin HLiao XShi XShao Z(2013)RTRM: A Response Time-Based Replica Management Strategy for Cloud Storage SystemGrid and Pervasive Computing10.1007/978-3-642-38027-3_13(124-133)Online publication date: 2013
https://doi.org/10.1007/978-3-642-38027-3_13
Grawinkel MSub TBest GPopov IBrinkmann A(2012)Towards Dynamic Scripted pNFS LayoutsProceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis10.1109/SC.Companion.2012.13(13-17)Online publication date: 10-Nov-2012
https://dl.acm.org/doi/10.1109/SC.Companion.2012.13
Hacker TSpringer JHowe B(2011)Turning the tide of the data delugeProceedings of the first annual workshop on High performance computing meets databases10.1145/2125636.2125644(21-24)Online publication date: 13-Nov-2011
https://dl.acm.org/doi/10.1145/2125636.2125644
Kato JIshikawa YIskra KHoefler T(2011)Design and implementation of parallel file aggregation mechanismProceedings of the 1st International Workshop on Runtime and Operating Systems for Supercomputers10.1145/1988796.1988802(33-40)Online publication date: 31-May-2011
https://dl.acm.org/doi/10.1145/1988796.1988802
Li XDong BXiao LRuan LDing Y(2011)Small Files Problem in Parallel File SystemProceedings of the 2011 International Conference on Network Computing and Information Security - Volume 0210.1109/NCIS.2011.143(227-232)Online publication date: 14-May-2011
https://dl.acm.org/doi/10.1109/NCIS.2011.143
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents