Semantic-based distributed i/o with the paramedic framework
Pages 175 - 184
Abstract
Many large-scale applications simultaneously rely on multiple resources for efficient execution. For example, such applications may require both large compute and storage resources; however, very few supercomputing centers can provide large quantities of both. Thus, data generated at the compute site oftentimes has to be moved to a remote storage site for either storage or visualization and analysis. Clearly, this is not an efficient model, especially when the two sites are distributed over a wide-area network.
Thus, we present a framework called "ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing" which uses application-specific semantic information to convert the generated data to orders-of-magnitude smaller metadata at the compute site, transfer the metadata to the storage site, and re-process the metadata at the storage site to regenerate the output. Specifically, ParaMEDIC trades a small amount of additional computation (in the form of data post-processing) for a potentially significant reduction in data that needs to be transferred in distributed environments.
References
[1]
S. F. Altschul, T. L. Madden, A. A. Schaffer, J. Zhang, Z. Zhang, W. Miller, and D. J. Lipman. Gapped BLAST and PSI-BLAST: A New Generation of Protein Database Search Programs. Nucleic Acids Research, 25:3389--3402, 1997.
[2]
T. Baer and P. Wyckoff. A Parallel I/O Mechanism for Distributed Systems. In Cluster, 2004.
[3]
San Diego Supercomputing Center. Parallel 3D FFT Library. http://www.sdsc.edu/us/resources/p3dfft.php.
[4]
A. Darling, L. Carey, and W. Feng. The Design, Implementation, and Evaluation of mpiBLAST. In International Conference on Linux Clusters: The HPC Revolution, 2003.
[5]
J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSDI, 2004.
[6]
I. Foster, D. Kohr, R. Krishnaiyer, and J. Mogill. Remote I/O: Fast access to distant storage. In Proceedings of the Fifth Workshop on I/O in Parallel and Distributed Systems, 1997.
[7]
Matteo Frigo and Steven G. Johnson. The design and implementation of FFTW3. Proceedings of the IEEE, 93(2):216--231, 2005.
[8]
M. Gardner, W. Feng, J. Archuleta, H. Lin, and X. Ma. Parallel Genomic Sequence-Searching on an Ad-Hoc Grid: Experiences, Lessons Learned, and Implications. In SC, 2006.
[9]
J. Lee, X. Ma, R. Ross, R. Thakur, and M. Winslett. RFS: Efficient and flexible remote file access for MPI-IO. In Cluster, 2004.
[10]
J. Lee, R. Ross, S. Atchley, M. Beck, and R. Thakur. MPI-IO/L: efficient remote i/o for mpi-io via logistical networking. In IPDPS, 2006.
[11]
TCP Linda. http://www.lindaspaces.com/products/linda_overview.html.
[12]
R. Thakur, W. Gropp, and E. Lusk. Data sieving and collective I/O in ROMIO. In Proceedings of the 7th Symposium on the Frontiers of Massively Parallel Computation, February 1999.
Index Terms
- Semantic-based distributed i/o with the paramedic framework
Recommendations
Semantics-based distributed I/O for mpiBLAST
PPoPP '08: Proceedings of the 13th ACM SIGPLAN Symposium on Principles and practice of parallel programmingBLAST is a widely used software toolkit for genomic sequence search. mpiBLAST is a freely available, open-source parallelization of BLAST that uses database segmentation to allow different worker processes to search (in parallel) unique segments of the ...
G-BLAST: a Grid-based solution for mpiBLAST on computational Grids
Over the past few years, research and development in bioinformatics (e.g. genomic sequence alignment) has grown with each passing day fueling continuing demands for vast computing power to support better performance. This trend usually requires ...
A trace-based study of SMB network file system workloads in an academic enterprise
SummerSim '20: Proceedings of the 2020 Summer Simulation ConferenceStorage system traces are important for examining real-world applications, studying potential bottlenecks, as well as driving benchmarks in the evaluation of new system designs. While file system traces have been well-studied in earlier work, it has ...
Comments
Information & Contributors
Information
Published In
June 2008
252 pages
ISBN:9781595939975
DOI:10.1145/1383422
- General Chairs:
- Manish Parashar,
- Karsten Schwan,
- Program Chairs:
- Jon Weissman,
- Domenico Laforenza
Copyright © 2008 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 23 June 2008
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
HPDC '08: International Symposium on High Performance Distributed Computing
June 23 - 27, 2008
MA, Boston, USA
Acceptance Rates
Overall Acceptance Rate 166 of 966 submissions, 17%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 221Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in