Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by ... more Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by the Los Alamos National Security, LLC for the National Nuclear Security Administration of the US Department of Energy under contract DE-AC52-06NA25396. By acceptance of ...
ABSTRACT The exponential growth of datasets has been well-known for many years and has been obser... more ABSTRACT The exponential growth of datasets has been well-known for many years and has been observed across a wide range of disciplines, including checkpoints for tightly-coupled parallel scientific simulations, genetic databases for bioinfor-matics research, and many others. With disk capacity rapidly outpacing disk bandwidth traditional methods of handling I/O may become ineffective. To address these trends we propose using a new architecture for parallel computing with multiple avenues for improvement of data management. To this end we analyze and demonstrate the performance benefits of one such avenue, namely taking advantage of information about upcoming tasks to asynchronously prestage data prior to job execution and present a discussion of the hardware necessary for implementing it in a real environment as well as a prototype implementation using the Makeflow workflow language and a custom hierarchical master-worker driver.
Browse by Title. MINDS@UW Home >; Browse by Title. Browse by Title. 0-9; A; B; C; D; E; F; G; ... more Browse by Title. MINDS@UW Home >; Browse by Title. Browse by Title. 0-9; A; B; C; D; E; F; G; H; I; J; K; L; M; N; O; P; Q; R; S; T; U; V; W; X; Y; Z. Or enter first few letters:Browse for items that begin with these letters. Sort by: title Order: ascending Results: 5. << Showing 9115-9134 of 18296. >>. Intelligent Routing Using Network Processors: Guiding Design Through Analysis. ��� Seshadri, Madhu Sudanan; Bent ...
ABSTRACT The I/O bottleneck in high-performance computing is becoming worse as application data c... more ABSTRACT The I/O bottleneck in high-performance computing is becoming worse as application data continues to grow. In this work, we explore how patterns of I/O within these applications can significantly affect the effectiveness of the underlying storage systems and how these same patterns can be utilized to improve many aspects of the I/O stack and mitigate the I/O bottleneck. We offer three main contributions in this paper. First, we develop and evaluate algorithms by which I/O patterns can be efficiently discovered and described. Second, we implement one such algorithm to reduce the metadata quantity in a virtual parallel file system by up to several orders of magnitude, thereby increasing the performance of writes and reads by up to 40 and 480 percent respectively. Third, we build a prototype file system with pattern-aware prefetching and evaluate it to show a 46 percent reduction in I/O latency. Finally, we believe that efficient pattern discovery and description, coupled with the observed predictability of complex patterns within many high-performance applications, offers significant potential to enable many additional I/O optimizations.
We present a study of six batch-pipeline scientific workloads that are candidates for execution o... more We present a study of six batch-pipeline scientific workloads that are candidates for execution on computational grids. Whereas other studies focus on the behavior of single applications, this study characterizes workloads composed of pipelines of sequential processes that use file storage for communication and also share measurements of the memory, CPU, and I/O requirements of individual components as well as analyses of I/O sharing within complete batches. We conclude with a discussion of the ramifications of these workloads for end-to-end scalability and overall system design.
Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by ... more Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by the Los Alamos National Security, LLC for the National Nuclear Security Administration of the US Department of Energy under contract DE-AC52-06NA25396. By acceptance of ...
ABSTRACT The exponential growth of datasets has been well-known for many years and has been obser... more ABSTRACT The exponential growth of datasets has been well-known for many years and has been observed across a wide range of disciplines, including checkpoints for tightly-coupled parallel scientific simulations, genetic databases for bioinfor-matics research, and many others. With disk capacity rapidly outpacing disk bandwidth traditional methods of handling I/O may become ineffective. To address these trends we propose using a new architecture for parallel computing with multiple avenues for improvement of data management. To this end we analyze and demonstrate the performance benefits of one such avenue, namely taking advantage of information about upcoming tasks to asynchronously prestage data prior to job execution and present a discussion of the hardware necessary for implementing it in a real environment as well as a prototype implementation using the Makeflow workflow language and a custom hierarchical master-worker driver.
Browse by Title. MINDS@UW Home >; Browse by Title. Browse by Title. 0-9; A; B; C; D; E; F; G; ... more Browse by Title. MINDS@UW Home >; Browse by Title. Browse by Title. 0-9; A; B; C; D; E; F; G; H; I; J; K; L; M; N; O; P; Q; R; S; T; U; V; W; X; Y; Z. Or enter first few letters:Browse for items that begin with these letters. Sort by: title Order: ascending Results: 5. << Showing 9115-9134 of 18296. >>. Intelligent Routing Using Network Processors: Guiding Design Through Analysis. ��� Seshadri, Madhu Sudanan; Bent ...
ABSTRACT The I/O bottleneck in high-performance computing is becoming worse as application data c... more ABSTRACT The I/O bottleneck in high-performance computing is becoming worse as application data continues to grow. In this work, we explore how patterns of I/O within these applications can significantly affect the effectiveness of the underlying storage systems and how these same patterns can be utilized to improve many aspects of the I/O stack and mitigate the I/O bottleneck. We offer three main contributions in this paper. First, we develop and evaluate algorithms by which I/O patterns can be efficiently discovered and described. Second, we implement one such algorithm to reduce the metadata quantity in a virtual parallel file system by up to several orders of magnitude, thereby increasing the performance of writes and reads by up to 40 and 480 percent respectively. Third, we build a prototype file system with pattern-aware prefetching and evaluate it to show a 46 percent reduction in I/O latency. Finally, we believe that efficient pattern discovery and description, coupled with the observed predictability of complex patterns within many high-performance applications, offers significant potential to enable many additional I/O optimizations.
We present a study of six batch-pipeline scientific workloads that are candidates for execution o... more We present a study of six batch-pipeline scientific workloads that are candidates for execution on computational grids. Whereas other studies focus on the behavior of single applications, this study characterizes workloads composed of pipelines of sequential processes that use file storage for communication and also share measurements of the memory, CPU, and I/O requirements of individual components as well as analyses of I/O sharing within complete batches. We conclude with a discussion of the ramifications of these workloads for end-to-end scalability and overall system design.
Uploads
Papers by john bent