I/O Requirements of Scientific Applications: An Evolutionary View
Evgenia Smirni, Ruth A. Aydt, Andrew A. Chien, Daniel A. Reed
Department of Computer Science
University of Illinois
Urbana, Illinois 61801
email:fesmirni,aydt,achien,reedg@cs.uiuc.edu
Abstract
The modest I/O configurations and file system limitations
of many current high-performance systems preclude solution
of problems with large I/O needs. I/O hardware and file system parallelism is the key to achieve high performance. We
analyze the I/O behavior of several versions of two scientific
applicationson the Intel Paragon XP/S. The versions involve
incremental application code enhancements across multiple
releases of the operating system. Studying the evolution
of I/O access patterns underscores the interplay between
application access patterns and file system features.
Our results show that both small and large request sizes
are common, that at present application developers must
manually aggregate small requests to obtain high disk transfer rates, that concurrent file accesses are frequent, and that
appropriate matching of the application access pattern and
the file system access mode can significantly increase application I/O performance. Based on these results, we describe
a set of file system design principles.
1. Introduction
For grand challenge scientific applications, where manipulating large volumes of data is a necessity, scalable
I/O performance is a critical requirement. Unfortunately,
rapid increases in computation and communication hardware performance have long outstripped the performance
improvements of secondary and tertiary storage devices.
This increasing disparity between the hardware performance
of individual processors and storage devices dictates use of
hardware and software I/O parallelism.
During the past decade, a wide variety of parallel I/O
systems have been proposed and built [2, 8, 11]. All these
systems exploit parallel I/O devices (partitioningdata across
disk arrays [1] for parallelism) and data management techniques (prefetching and write-behind) in an attempt to de This work was supported in part by the Advanced Research Projects
Agency under ARPA contracts DAVT63-91-C-0029 and DABT63-93-C0040, by the National Science Foundation under grant NSF ASC 92-12369,
and by the Aeronautics and Space Administration under NASA Contracts
NGT-51023 and USRA 5555-22. Andrew Chien is supported in part by an
NSF Young Investigator Award CCR-94-57809.
liver high I/O performance. However, parallel file system
experience has shown that performance is critically sensitive
to the distribution of data across storage devices and that no
single file management policy yields good performance for
all application access patterns. As a result, many parallel I/O
systems have high peak performance, but their much lower
delivered performance continues to constrain the domain of
I/O intensive applications.
Although there have been extensive studies of uniprocessor file access patterns, much less is known about the
I/O requirements of scalable scientific applications. In particular, there is no clear consensus on the complexity and
diversity of file access patterns and resource demands of parallel scientific applications. As a consequence, parallel file
system designers have often been forced to extrapolate from
measurements on uniprocessors or vector supercomputers.
Neither of these environments is characteristic of parallel
application I/O needs on scalable parallel systems.
Clearly, understanding the interactions between application I/O request patterns and the hardware and software of
parallel I/O systems is a precursor to designing more effective I/O management policies. As part of the Scalable I/O
Initiative (SIO) [14], our twin goals are to collect detailed
performance data on the I/O characteristics and access patterns of a variety of scalable parallel scientific applications
and to use this information to design and evaluate parallel
I/O file system management policies.
In this paper, we report an analysis of the evolving I/O
behavior and performance of two SIO codes, an electron
scattering application [18] and a computational fluid dynamics application [4]. Over eighteen months, we studied the
evolving resource demands of both applications on the Intel
Paragon XP/S and captured the interplay between application resource demands and system responses. By tracking
the evolution of each application’s I/O demands as the code
was tuned, we isolated and identified file access patterns
that were intrinsic to the applications and not artifacts of the
developers’ performance optimizations with respect to the
file system features.
The remainder of the paper is organized as follows. In
x2, we outline related work in parallel I/O characterization.
This is followed in x3 by a brief description of our experimental methodology using the Pablo performance analysis
environment and a summary of parallel file system characteristics. In x4–x5, we describe the evolution of two I/O
intensive codes, followed in x6 by a comparison of their I/O
behaviors. Finally, x7 summarizes our findings and outlines
future work.
2. Related Work
Several I/O characterization efforts for scientific applications have concentrated on the I/O behavior of vector
supercomputers [9, 12, 13]. Miller and Katz [9] measured
a workload of mostly computational fluid dynamics applications on a Cray Y-MP and characterized the I/O behavior
as highly regular, cyclical, and bursty. They first proposed
a high-level I/O classification based on compulsory, checkpoint, and data staging I/O operations.
Pasquale and Polyzos [12] used clustering and regression
analysis to examine I/O demands and their relationship to the
elapsed computation time of the Cray Y-MP workload at the
San Diego Supercomputing Center (SDSC). They also isolated and analyzed two I/O intensive applications on a Cray
90 at SDSC [13]. By focusing on both physical resource
usage and functional application composition, Pasquale and
Polyzos concluded that supercomputer I/O behavior was recurrent and predictable, based on the iterative nature of many
scientific applications.
In contrast to the results on vector systems, our more
recent analysis on the Intel Paragon XP/S [3] indicated that
there are large variations in the temporal and spatial access
patterns of scientific applications. Some cyclic behavior was
noted, but the applications’ I/O patterns were more irregular,
with both extremely small and extremely large requests. Our
work complements this earlier study by isolating the reasons
that application developers opt to use parallel file systems
in particular ways.
Kotz and Nieuwejaar [7] examined application file access
characteristics on the Intel iPSC/860’s Concurrent File System at NASA’s Ames Research Center. In a related study,
Parakayastha et al [15] investigated file access patterns on a
CM-5 under the Scalable File System (SFS) at the National
Center of Supercomputing Applications. The results of both
studies showed that the majority of the file accesses were
small, even though the vast majority of data was transferred
via a few large requests. Subsequent comparison of the file
access characteristics on both systems [10] indicated that
users “tune” their applications to the underlying parallel system’s idiosyncrasies in an effort to maximize performance.
Our work builds on these earlier studies by examining the
interplay between file system features and application developer decisions.
3. Experimental Approach
In contrast to previous I/O characterization studies that
considered a single version of a code, we have tracked the
evolving I/O behavior of two applications over an eighteen
month period. The two applications are both part of the
Scalable I/O application suite, they are in active use by
their developers, and they are evolving rapidly based on
continuing changes to the Intel Parallel File System (PFS).
To capture traces of application I/O requests and parallel file system responses, we used an extended version of
the Pablo performance analysis environment [17]. All data
was captured on the 512-node Intel Paragon XP/S at the
Caltech Center of Advanced Computing Research. Below,
we describe the I/O analysis mechanisms and the salient
characteristics of the Intel parallel file system.
3.1. Pablo Software Infrastructure
Building on the lessons of several previous performance
analysis toolkits, Pablo [16, 17] is a portable performance
environment that supports both performance data capture
and analysis. The Pablo instrumentation software captures
dynamic performance data via instrumented source code
that is linked with a data capture library. During program
execution, the instrumentation code generates performance
data that can either be directly recorded by the data capture
library or processed by one or more data analysis extensions
prior to recording. After program execution completes, the
data can be analyzed with a toolkit of data transformation
modules and a graphical programming model that allows
users to interactively connect and configure a data analysis
graph.
Detailed I/O event traces include the time, duration, size,
and other parameters for each I/O operation. Statistical
summaries can take one of three forms: file lifetime, time
window, or file region. File lifetime summaries include
the number and total duration of file reads, writes, seeks,
opens, and closes, as well as the number of bytes accessed
for each file, and the total time each file was open. Time
window summaries contain similar data, but allow one to
specify a window of time for summarization. Finally, file
region summaries are the spatial analog of time window
summaries; they define a summary over the accesses to
a file region. Collectively, the raw event traces and the
statistical summaries provide a detailed view of application
I/O request patterns and file system responses.
3.2. Intel Paragon XP/S Configuration
Interpreting the results of any empirical study is critically
dependent on understanding the experimental environment.
All our experiments were conducted on the Caltech 512node Intel Paragon XP/S, organized as a 16x32 mesh. For
all experiments, sixteen I/O nodes were used, each with a
4.8GB RAID-3 disk array, and files were striped across the
disk arrays in units of 64K bytes, the PFS default.
PFS offers several file access modes, each designed to
support a specific class of application I/O patterns. The
default I/O mode is M UNIX, which provides standard UNIX
file sharing semantics when more than one process access the
same file. In this mode, each process has a unique file pointer
and can access information anywhere in the file. There are
no restrictions on the range of possible file request sizes.
Because request atomicity is preserved, the performance of
mode M UNIX can be poor when multiple processes access
the same file. In that case, it is often more efficient to use
the M RECORD or M ASYNC modes.
With the M RECORD mode, there is a unique file pointer
for every process, all nodes must access fixed size records,
and concurrent operations occur in node order. In this way,
each process can operate on separate file areas in a parallel
and highly structured fashion.
In the M ASYNC mode, every process has its own file
pointer and, in contrast to M RECORD, variable length accesses are possible. Atomicity is not preserved and processes need not read from or write to the same file concurrently.
The M GLOBAL mode is often used for initial compulsory
reads, where all processes must access the same data. With
this mode, a shared file pointer is maintained, all processes
must access the same data in a synchronized fashion, and
data are read/written only once. Intuitively, only one of
a group of identical I/O operation occurs, and the data is
shared among all the processes.
With the M SYNC mode, all processes share a single file
pointer and I/O requests occur in node order. All I/O requests
are synchronized, but the request size can vary from node to
node.
Finally, with the M LOG mode all processes share a common file pointer, I/O requests are processed first-come-firstserve, they are not synchronized, and the request size can
vary across processes. As the name suggests, this mode is
normally used to access stdin, stdout, and stderr
files.
4. Schwinger Multichannel Electron Scattering
The study of low-energy electron-molecule collisions is
of interest in many contexts, including aerospace applications, atmospheric studies, and the processing of materials
using low-temperature plasmas (e.g., semiconductor fabrication). The Schwinger Multichannel (SMC) method is
an adaptation of Schwinger’s variational principle for the
scattering amplitude that makes it suitable for calculating
low-energy electron-molecule collisions [18].
The scattering probabilities are obtained by solving linear systems whose terms must be evaluated by numerical
quadrature. Generation of the quadrature data is computationally intensive, and the total quadrature data volume
is highly dependent on the nature of the problem. The
quadrature data is formulated in an energy independent way,
making it possible to solve the scattering problem at many
energies without quadrature data recalculation. Because the
quadrature data is too voluminous to fit in the individual
processor memory, an out of core solution is required.
ESCAT is a parallel implementation of the Schwinger
Multichannel method written in C, FORTRAN, and assembly language [18]. From an I/O perspective, the code has
four distinct execution phases:
Phase One: Initializationdata is read from three input
files (compulsory I/O). During this phase, the problem
definition and some initial matrices are loaded.
Phase Two: Quadrature data is written to disk (data
staging). In this phase, all nodes participate in the
calculation and storage of the requisite quadrature
data set. This phase is composed of a series of compute/write cycles, with the write steps synchronized
among the nodes. The number of data files written
corresponds to the number of collision channels.
Phase Three: Quadrature data is read from disk (data
staging). This phase involves calculations that depend
on the collision energy. Energy dependent data structures are generated and combined with the reloaded
quadrature data.
Phase Four: Results of the calculations are written
to disk (compulsory I/O). The number of output files
corresponds to the number of collision channels.
As noted in x3, we used the Pablo performance environment’s I/O instrumentation software to capture the ESCAT
code’s I/O behavior, including the individual request sizes,
their timings, and their temporal and spatial patterns. Based
on our analysis of this data, we offered several suggestions to
the code’s developers, many of which were subsequently implemented. Furthermore, the Intel PFS designers modified
the Paragon XP/S file system to better meet the application
I/O needs and improve I/O performance. Below, we analyze
the evolution of ESCAT application I/O requirements for a
specific test problem.
4.1. I/O Requirements
As a baseline for I/O performance analysis, we considered a representative, though modest problem, taken from a
study of electronic excitation of ethylene to its first triplet
state. For this problem, two collision channels are of interest: the elastic-scattering channel and the inelastic tripletexcitation channel.
Using the ethylene data set, we measured the performance and behavior of six distinct versions of the ESCAT
code on 128 processors during a period of eighteen months 1 .
1 Although
we have collected and analyzed instrumentation data from
two much larger problems, the electronic excitation of carbon monoxide
and the elastic scattering cross section for boron trichloride, the focus of
this paper is the smaller ethylene data set.
Phase One
Phase Two
Phase Three
Phase Four
Version A
Version B
Version C
OSF 1.2, Pablo Beta
OSF 1.2, Pablo 4.0
OSF 1.3, Pablo 4.0
I/O Activity
All Nodes
Node zero
Node zero
Node zero
I/O Mode
M UNIX
M UNIX
M UNIX
M UNIX
I/O Activity
Node zero
All Nodes
All Nodes
Node zero
I/O Mode
M UNIX
M UNIX
M RECORD
M UNIX
I/O Activity
Node zero
All Nodes
All Nodes
Node zero
I/O Mode
M UNIX
M ASYNC
M RECORD
M UNIX
Table 1: Node activity and file access modes (ESCAT)
During this interval, we considered the effects of operating
system changes, new application code versions, and software instrumentation updates. Thus, we have a detailed
record of the performance pitfalls that triggered application
and operating system evolution.
Figure 1 illustrates the progressive reduction in total execution time for ESCAT with the ethylene data set. Overall,
total execution time was reduced by 20 percent from the first
ESCAT version (version A in Figure 1) to the final version
(version C in Figure 1). Although this reduction may seem
modest, the ESCAT code’s total execution time is not dominated by I/O for this small problem. For larger problems,
the optimizations directly translate to order of magnitude
performance improvements.
6800
A
Execution Time
6600
6400
B
6200
6000
Operation
open
gopen
read
seek
write
iomode
close
Operation Time 100
Total I=O Time
A
53.68
–
42.64
1.01
1.27
–
1.39
B
0.00
4.05
0.24
63.21
28.75
2.94
0.81
C
0.03
21.65
1.53
1.75
55.63
16.06
3.34
Table 2: Aggregate I/O performance summaries (ESCAT)
5800
5600
Node zero then collects the quadrature data and writes it to
intermediate staging files on disk. In phases three and four,
node zero reads and writes all data.
Table 2 shows that for version A, most of the I/O time
is spent opening and reading files. Because the default
M UNIX mode is used, all reads during phase one are serialized. This inefficiency makes reads a high fraction of the
total I/O time. However, because all writes occur through
node zero, there is no contention, and the time spent on
writes and seeks is negligible.
C
5400
ESCAT Executions
Figure 1: Execution time for six ESCAT code progressions
For brevity’s sake, our analysis will concentrate on code
versions A, B, and C. Table 1 summarizes the major differences among the three versions: the number of nodes that
actively participate in I/O during each of the four application
phases and the PFS I/O mode used in each phase. Similarly,
Table 2 shows the fraction of time attributable to each I/O
operation type.
The I/O activity for version A reflects the influence of
the Concurrent File System (CFS) on the Intel Touchstone
Delta, where the code was first developed. As Table 1 shows,
in version A all nodes concurrently open and read the initialization files in phase one. In phase two, all nodes repeatedly
compute, synchronize, and compose the quadrature data.
In version B, a significant reduction in read time was
achieved via code restructuring. In this version, node zero
reads the input data and broadcasts it to the remainder of the
nodes; see Table 1. In phases two and three, all nodes participate in writing/reading the quadrature data to/from disk.
To eliminate the time spent opening the same file concurrently by multiple nodes, a global open operation (gopen)
is used as an alternative to the more expensive open operation. In phase two, to simplify reloading the quadrature
data, each node seeks to a calculated offset dependent on
the node number, iteration, and the Paragon PFS stripe size
before writing any data. Intel’s M UNIX file mode is used
for these writes, and seek time dominates the total I/O time.
The use of M RECORD mode to reload the quadrature data in
phase three further contributes to the reduction in read time
from version A;see Table 2.
As Table 1 shows, the code structure of version C is similar to that for version B, the only difference is the file access
mode of phase two. Due in part to our recommendation,
The large read sizes are a direct effect of the programmer’s choice to use M RECORD to access the quadrature data
in phase three; to guarantee good performance when using
M RECORD, the request size must be a multiple of the stripe
size. Changes in the CDFs for writes from version A to
versions B and C are not dramatic; all write requests are
small, and most are less 2K bytes.
fraction of reads
fraction of data
A
B, C
A
B, C
1
10
100
1000 10000 100000 1e+06
CDF
(a) Read Size (bytes)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
fraction of writes
fraction of data
A, B, C
A
B, C
0
500
1000
1500
2000
2500
3000
(b) Write Size (bytes)
Figure 2: CDF of read/write request sizes and data transfers
for ESCAT
Intel introduced the more efficient M ASYNC mode in the
OSF/1 1.3 operating system release. This file access mode
dramatically reduces seek and write times. Recall that with
M ASYNC every process has its own file pointer and operation atomicity is left to the discretion of the programmer.
Consequently, the system overhead for preserving operation
atomicity is avoided.
4.3. Temporal I/O Behavior
Spatial access patterns are but a part of the story; the
temporal spacing of request sizes, operations, and use of
file system access modes also affects performance. Figure 3 depicts read request sizes as a function of program
execution time for versions A and C.2 In both cases, read
activity occurs only near the beginning and the end of the
ESCAT execution. First, initialization files are read. Because only node zero reads data in version C, there is a
significant reduction in the density and span of initial read
time between versions A and C. In the final read phase,
where the quadrature data is transferred from disk, the read
request sizes change significantly. In version C, all nodes
read the quadrature with 128KB increments (twice the PFS
stripe size), whereas in version A, node zero reads the data
in small chunks (less than 2K bytes) and broadcasts them to
the other nodes.
1e+06
Read Size (bytes)
CDF
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
10000
1000
100
10
1
0
1000 2000 3000 4000 5000 6000 7000
Execution Time
4.2. I/O Request Sizes
1e+06
Read Size (bytes)
Apart from changes in the code structure and file access
modes, the I/O request sizes across the three ESCAT versions also differ. Figure 2 shows the cumulative distribution
function (CDF) of the percentage of reads and writes versus
the request size for the three code versions, as well as the
fraction of data transferred by each request size. Because
there are no changes in request sizes from version B to version C, the CDFs for these two versions are shown in the
same plot.
The number of small reads (i.e., those less than 2K bytes)
declines dramatically from version A to versions B and C.
In version A, 97 percent of all read operations are small,
though they transfer only 40 percent of the data. In the two
following versions, only 50 percent of reads are small, and
large 128KB reads (two PFS file stripes) transfer 98 percent
of the data read.
Version A
100000
Version C
100000
10000
1000
100
10
1
0
1000
2000
3000
4000
5000
6000
Execution Time
Figure 3: File read sizes for versions A and C of ESCAT
The differences in the application temporal behavior
2 The data for version B is not shown, because it is very similar to that
for version C.
Write Size (bytes)
3000
Seek Duration (seconds)
5
4
3
2
1
0
1000 2000 3000 4000 5000 6000 7000
Execution Time
0.45
Version C
0.4
0.35
0.3
0.25
0.2
0.1
0.05
2000
0
0
1500
1000
2000
3000
4000
5000
6000
Execution Time
1000
Figure 5: Seek operation durations for versions B and C of
ESCAT
500
0
0
1000 2000 3000 4000 5000 6000 7000
Execution Time
eliminates the need for serialization and clearly improves
performance.
3000
Write Size (bytes)
Version B
7
6
0.15
Version A
2500
9
8
0
Seek Duration (seconds)
across the three versions are the effect of different file access
modes, number of concurrent accesses, and corresponding
request sizes. In version A, all nodes concurrently access
the initialization files using the M UNIX mode, serializing
all requests. Because only node zero accesses the initialization files in version C, the duration and and total number of
initial reads declines precipitously.
Figure 4 shows the write request sizes for versions A
and C; version B is similar to version C. As with reads,
the write structure of both versions has two distinct phases.
First, the quadrature data is generated and transferred to
disk. In version A, node zero coordinates these writes with
four different request sizes. In version C, all write requests
are of the same size, and the individual nodes write the
data directly using the M ASYNC mode. Finally, in all three
versions, node zero writes the results of computation to the
disk.
4.4. Discussion
Version C
2500
2000
1500
1000
500
0
0
1000
2000
3000
4000
5000
6000
Execution Time
Figure 4: File write sizes for versions A and C of ESCAT
Figure 5 depicts the seek durations for versions B and
C. Notice the difference in the order of magnitude on the
y-axis for the plots of Figure 5. The advantage of M ASYNC
versus M UNIX is further illustrated in this figure; seek times
are almost eliminated. Recall that M UNIX provides the
standard UNIX file sharing semantics by serializing requests
when more than one process access the same file. In this
case, each node has a unique file pointer and seeks prior to
reading or writing data in the file. The use of M ASYNC
The I/O behavior of the ESCAT evolution highlights the
importance of application tuning to achieve high I/O performance. The most critical I/O demands are placed on the
scratch files that contain the intermediate quadrature data.
These demands are proportional to the number of total collision outcomes.
Table 3 shows that for the ethylene case, where just two
collision outcomes are of interest, only 0.73 percent of the
total execution time is spent on I/O for the optimized version
C. At first glance, it appears that I/O is of little import
because this code is heavily computation bound. While this
is true for the ethylene data set, it is not true for larger data
sets. The complexity of the quadrature data volume grows
as O(N 3 ), where N is the number of electron scattering
outcomes. Thus, gains from I/O performance optimization
are more pronounced as the number of collision outcomes
increases.
Using version C, the code developers can now attack
larger, more complex problems. Table 3 shows the I/O
costs for a larger carbon monoxide data set, where 13 collision outcomes are of interest. In this case, the total time
spent on I/O is twenty percent of total execution time after
Operation Time
Total Execution Time
9500
A
Carb. Mon.
(256 nodes)
C
0.00
7.45
9.50
0.00
0.03
–
2.41
19.40
Table 3: Percentage of total execution time by I/O operation
type for ESCAT
optimization3.
In summary, the application developers were able to restructure their code, tuning request sizes to match the PFS
file stripe size. In addition, they opted to substitute concurrent reads by all nodes with single node read and broadcast
mechanisms, despite of the introduced complication in the
code structure. Request aggregation and prefetching by the
file system would simplify code structure and eliminate the
need for code restructuring to exploit file system characteristics.
Execution Time
Operation
open
gopen
read
seek
write
iomode
close
All I/O
Ethylene
(128 nodes)
A
B
C
1.60 0.00 0.00
– 0.19 0.16
1.27 0.01 0.01
0.03 2.91 0.01
0.04 1.32 0.41
– 0.14 0.12
0.04 0.04 0.02
2.97 4.60 0.73
100
Phase One: Three input files are used to initialize the
system (compulsory I/O). The first file contains parameters such as the Reynolds number, the number of
3 For detailed I/O analysis of the carbon monoxide data set see
http://www-pablo.cs.uiuc.edu/Projects/IO/sioDir/escat/escat.html
8500
B
8000
7500
C
7000
Prism Code Versions
Figure 6: Execution time for three PRISM code versions
mesh elements, the coordinates for each element, the
number of curved edges, and the boundary conditions
for each edge. The second file is a restart file that
contains the initial conditions. The third input file is
the connectivity file that contains information used to
establish the internal boundary system of the spectral
element mesh.
5. 3-D Numerical Simulation of the NavierStokes Equations
Computational fluid dynamics is an increasingly frequent
alternative to experimental study, particularly for understanding high-speed turbulent flow. PRISM is a parallel
implementation of a 3-D numerical simulation of the NavierStokes equations written in C [4]. The parallelization is
achieved by apportioning slides of the periodic domain to the
nodes, with a combination of spectral elements and Fourier
modes used to investigate the dynamics and transport properties of turbulent flow.
This code models a geometry where the flow is periodic
in at least one direction (e.g., flow past a cylinder, flow in a
channel, and flow over a backward-facing step). An initial
velocity field is given by the input data, and the solution is
integrated forward in time by numerically solving the equations that describe advection and diffusion of momentum in
the fluid. From an I/O perspective, there are three distinct
execution phases:
9000
Phase Two: The integration function for the NavierStokes simulation integrates the status of the fluid
forward in time from its current to its new state. History points are written to disk during the integration
(checkpointing). A measurement file is written by
node zero that includes lift and drag forces, viscous
forces, and the kinetic energy of each mode. Flow
statistics for velocity, vorticity, and turbulent stresses
are written to three separate files. Each of these files
contains the mean, variance, skewness, and flatness
associated with each field.
Phase Three: During the final or postprocessing
phase, results are transformed back to physical space
and the field file is written to disk (compulsory I/O).
As with the ESCAT code, we instrumented PRISM using
the Pablo performance environment. Below we describe the
code evolution and I/O characterization of PRISM.
5.1. I/O Requirements
We tracked the evolution of the PRISM code using a
small test problem, and captured performance data on 64
nodes of the 512 node Intel Paragon XP/S at Caltech. The
test problem consisted of 201 mesh elements and a Reynolds
number of 1000. The model was simulated for 1250 time
steps, with checkpointing every 250 time steps (i.e., a total
of five checkpoints). Because our experience with PRISM
is somewhat less extensive, we analyzed only three versions
of the code, all executed under OSF/1 R1.3.
Table 4 summarizes the I/O modes and activities for the
Version A
OSF 1.3, Pablo 4.0
I/O Activity I/O Mode
Version B
OSF 1.3, Pablo 4.0
I/O Activity I/O Mode
Version C
OSF 1.3, Pablo 4.0
I/O Activity I/O Mode
Phase One
All Nodes
All Nodes
All Nodes
Phase Two
Phase Three
Node Zero
Node Zero
P: M UNIX
R: M UNIX
C: M UNIX
M UNIX
M UNIX
Node Zero
All Nodes
P: M GLOBAL
R(h): M GLOBAL
R(b): M RECORD
C: M GLOBAL
M UNIX
M ASYNC
Node Zero
All Nodes
P: M GLOBAL
R: M ASYNC
C: M GLOBAL
M UNIX
M ASYNC
Table 4: Node activity and file access modes (PRISM)
three code versions4 , Figure 6 shows the twenty-three percent reduction in total execution time across the versions,
and Table 5 shows the effects of the different I/O mode
choices on application performance. As with ESCAT, application I/O optimization substantially reduced I/O time.
Comparison of the high level I/O characteristics of versions A of ESCAT and PRISM (see Tables 1 and 4) shows
marked similarities. Both codes use standard UNIX I/O operations, and all nodes read data in the initial, compulsory
I/O phase. For the other two PRISM phases, all I/O is administered though node zero. As Table 5 shows, the majority
of the PRISM I/O time for version A is spent opening and
reading files. In phases two and three, node zero writes the
measurement file, the statistics files, and the field file using
the M UNIX mode. Because only one node writes, there is
little I/O contention and the seek and write operations are
efficient.
After code restructuring, the read and file open times are
substantially lower for version B. The PRISM developers
preferred not to perform all reads through node zero, but
instead used the M GLOBAL access mode that aggregates
identical I/O requests, yielding a single disk I/O. In version
B, the parameter and connectivity files are accessed using the
M GLOBAL mode. Two access modes are used for the restart
file; its header is accessed using the M GLOBAL mode and
its body using the M RECORD mode. With the M RECORD
mode, every node is synchronized to read a specific area
in the file. As Table 5 shows, the write time in version B
increases as a consequence of the concurrent writes by all
processors to the field file of phase three.
In both versions A and B, the open operation is very
expensive. As with ESCAT, open times are reduced by using
the more efficient gopen operation in version C. Because it
also sets the file mode, the gopen call eliminates expensive
file mode operations.
Although the open time drops sharply in version C, the
read time increases significantly. In version C, to reduce
the time required to access the body of the restart file in
phase one (the body of the restart file is accessed via a few
4 In Table 4, the three input files in phase one are denoted by the first
letter of the file name.
requests of 155,584 bytes each), the programmer opted to
disable any system I/O buffering. By disabling buffering,
the read time of the header of the restart file (that is accessed
via a few requests of less than 40 bytes each) augmented
disproportionately. If I/O buffering were disabled after accessing the header of the restart file, read time would have
been as low as in version B, and a significant improvement
in the total I/O time would have been achieved. We will
return to this point below, where we more closely examine
the read behavior across the program execution timeline.
Operation
open
gopen
read
seek
write
iomode
flush
close
Operation Time 100
Total I=O Time
A
75.43
–
16.24
3.87
1.83
–
–
2.63
B
57.36
–
9.47
1.22
9.91
17.75
–
4.50
C
3.36
3.42
83.92
0.40
6.51
–
0.06
2.32
Table 5: Aggregate I/O performance summaries (PRISM)
5.2. I/O Request Sizes
Figure 7 shows the cumulative distribution function for
reads and writes for the three PRISM versions. There is
no significant variation in the access sizes across the three
versions. In version C, the connectivity file is read as binary
rather than text data, reducing the number of small reads.
In general, there are a large number of small (less than 40
bytes) read and write requests, although a few large requests
(greater 150KB) constitute the majority of I/O data volume.
5.3. Temporal I/O Behavior
Figure 8 shows that there is significant variation in the
temporal distribution of read requests during execution of the
three code versions. First, from version A to version B, the
total read time decreases by 125 seconds. This decrease is
an immediate effect of using the M GLOBAL and M RECORD
1e+06
Read Sizes (bytes)
CDF
1
fraction of
0.9 reads (A,B)
0.8
0.7
fraction of reads (C)
0.6
0.5
fraction of data (A,B)
0.4
0.3
0.2
fraction of data (C)
0.1
0
1
10
100
1000 10000 100000 1e+06
Version A
100000
10000
1000
100
10
1
0
50
(a) Read Size (bytes)
1
150
200
250
fraction of writes (A,B,C)
0.6
0.4
0.2
fraction of data(A,B,C)
Read Sizes (bytes)
1e+06
0.8
CDF
100
Execution Time
Version B
100000
10000
1000
100
10
0
1
10
100
1
1000 10000 100000 1e+06
0
(b) Write Size (bytes)
20
40
60
80
100
120
140
Execution Time
Figure 7: CDF of read and write request sizes and data
transfers for PRISM
5.4. Discussion
For the test problem we tracked, the PRISM code is not
as I/O intensive as ESCAT. However, as with ESCAT, the
I/O scalability of the PRISM code is crucial to solving larger
problems. Moreover, analysis of the PRISM code versions
highlights the need to efficiently support a range of request
sizes. Although current parallel systems favor large requests
because high bandwidths are achieved through parallelism,
low latency access for small requests is also essential.
As illustrated in version C of PRISM, a few small reads
can dominate overall I/O time. Robust I/O operations that
Read Sizes (bytes)
modes instead if the M UNIX mode, to access the three
initialization files.
In version A, the developers used standard UNIX I/O for
reads, resulting in serialization of the I/O operations seen
in Figure 8. An indirect effect of using the M GLOBAL and
M RECORD modes is node synchronization and compact
distribution of read requests across the execution timeline in
version B. Moving from version B to version C, PRISM’s
execution timeline becomes longer due to the disabling of
system I/O buffering when accessing the restart file.
Because most writes occur through node zero, the temporal write patterns and sizes differ little across the three
PRISM versions. Figure 9 shows the temporal write patterns for version C, where the five checkpoints are clearly
visible.
1e+06
Version C
100000
10000
1000
100
10
1
0
20
40
60
80 100 120 140 160 180
Execution Time
Figure 8: File read sizes for for three versions of PRISM
employ caching or prefetching are an attractive and less
confusing alternative to manual request aggregation. A file
system that dynamically tunes its policy to match the requirements of the application access patterns and disk performance characteristics is a promising alternative [6].
6. Application Comparisons
Although the ESCAT and PRISM codes differ dramatically in their algorithmic approaches, they share many common I/O characteristics and problems. First, small code
changes (e.g. a few I/O calls) can produce large changes
in I/O performance. This is both heartening and dismaying.
Wholesale code changes are not necessary to improve I/O
performance, but selecting inappropriate I/O routines can
severely affect achieved performance.
Version C
Write Size (bytes)
1e+06
100000
10000
1000
100
10
1
0
1000 2000 3000 4000 5000 6000 7000 8000
Execution Time
Figure 9: File write sizes for version C of PRISM
Second, tuning application I/O request sizes to match the
underlying parallel file system is necessary to achieve acceptable parallel I/O transfer rates. However, understanding
when and how to aggregate requests is possible only if one
understands the performance implications and implementation of the parallel I/O system. In short, optimizations are
closely tied to the idiosyncrasies of the parallel I/O system.
Third, both applications exhibit the same general I/O
structure of three distinct phases. In the first phase, initialization data is read from a small number of files (compulsory I/O). During the second phase, all nodes compute
and synchronize, and most data output occurs. PRISM uses
checkpointing, producing periodic bursts of I/O activity, and
ESCAT employs data staging for its out-of-core computations. In the last phase, both codes write final results.
In each of the three phases, I/O activity can be classified
across three dimensions: I/O request size, I/O parallelism,
and I/O access modes. To clearly establish similarities in the
evolution of both codes, we compare the initial access patterns of both applications and the optimized patterns of the
final application versions as influenced by the idiosyncrasies
of PFS.
6.1. Initial Access Patterns
In the initial version of both codes, at least 98 percent
of all reads were small (i.e., less than 1K bytes), although
the vast majority of data is read via a small number of large
requests (i.e., greater than 128K bytes). In both codes,
small writes predominate. This distribution of request sizes
reflects the application codes’ “natural” I/O needs (i.e., patterns that are natural and intuitive to the application developer based on the code structure and algorithm). Moreover,
only standard UNIX I/O calls were used. Conversations with
the application developers revealed that they opted for the
easiest and most natural implementation for their I/O [19, 5].
Both codes relied on a single node to coordinate parallel read and write operations by all nodes. This partitioning of I/O tasks across nodes is partially an artifact of the
codes’ previous platforms and partially a consequence of a
restricted set of I/O modes. Because there was no equivalent
to the M GLOBAL mode on the Intel Delta, where ESCAT
was developed, or the Intel iPSC/860, where PRISM was
developed, concurrent reads using the M UNIX were the
easiest and most natural choice [19]. Likewise, both codes
would have benefited from concurrent, synchronized writes
by all nodes, but in the first release of PFS no such I/O mode
was available, and both developers opted to to perform all
writes through node zero [19, 5].
6.2. Optimized Access Patterns
Because there were clear research benefits, both code
developers worked assiduously to optimize their application
I/O behavior and to exploit Intel PFS features. Most changes
optimized the read request sizes.
For ESCAT, the number of small requests, those less
than 1K bytes, dropped substantially. After optimization,
about 45 percent of reads were large (i.e., 128K bytes) and
equal to twice the PFS stripe size, effectively exploiting
the data striping of the PFS file system. Moreover, the
majority of data (98 percent) is now transferred via these
reads. Similarly, the PRISM developer reduced the fraction
of small reads, by using a binary data format.
To minimize the time consumed by reading initialization
data, both application developers changed the file access
scheme, albeit in different ways. In ESCAT, node zero
now reads the initialization files and broadcasts the data to
the other nodes. In PRISM, the same solution is realized
implicitly by opening the files using the Intel M GLOBAL
and M RECORD modes.
Writes were optimized in ESCAT by using the M ASYNC
mode, eliminating costly file access serialization; the application is then responsible for ensuring that no two nodes
concurrently write to the same file region. PRISM uses
M UNIX for the writes because they are still administered
through node zero.
Overall, the analysis of the three code versions for both
ESCAT and PRISM revealed many strengths and weaknesses of PFS. PFS achieves high transfer rates for large
request sizes that are multiples of the file stripe size. However, the performance for small requests is quite low 5. PFS
provides collective and non-collective operations that cover
a large spectrum of application needs. In addition, PFS
offers a wide variety of synchronous and asynchronous operation modes. However, in some cases, significant performance degradation is caused by unexpected behavior of a
file access mode, or perhaps inappropriate use of that mode.
7. Conclusions and Future Work
Characterization studies are by their nature inductive,
covering only a small sample of the possibilities. The two
5 To redress these limitations, Intel is developing optimizations to support small accesses.
applications we have studied represent but a few cases from
a large space of parallel applications. We believe they are indicative, though they are by no means an exhaustive description of the parallel I/O requirements or behavior exhibited
by scalable applications. In this work, we tried to isolate the
influence of available technology on the application code.
Doubtless, our performance results are conditioned by I/O
hardware, system software, and machine configurations.
We conclude that there are many opportunities for improving the performance of current parallel file systems.
Parallel I/O standardization is an important step that will
greatly contribute to code scalability and portability across
different architectures. The need for asynchronous and
collective operations is imperative. Request aggregation,
prefetching, and write behind are possible approaches.
Recognition and parallelization of I/O operations by the
compiler will enable the system to reorder and/or cluster
I/O requests and will provide substantial leverage to the application developer by reducing the unavoidable burden of
code tuning to exploit file system idiosyncrasies.
We are currently broadening our characterization studies. A study of a larger set of applications is underway.
Additionally, we plan to examine the effects of different
machine configurations (e.g., number of I/O nodes) and different architectures on I/O performance. From these characterizations, a comprehensive set of parallel file system I/O
benchmarks will be derived.
Acknowledgments
Phyllis Crandall contributed to the early evaluation of I/O
behavior of ESCAT and PRISM, and Chris Elford’s analysis of code behavior was invaluable. The electron scattering
code (ESCAT) was provided by Carl Winstead and Vincent
McKoy at Caltech. The PRISM code was obtained from
Ron Henderson and Dan Meiron at Caltech. All data presented here were obtained from code executions on the Intel
Paragon XP/S at the Caltech Center of Advanced Computing
Research.
References
[1] Chen, P. M., Lee, E. K., Gibson, G. A., Katz, R. H., and
Patterson, D. A., “RAID: High-performance, Reliable Secondary Storage”, ACM Computing Surveys, Vol. 26(2), pp.
145-185, June 1994.
[2] Corbett, P., Feitelson, D. G., Prost, J.-P., and Baylor, S. J.,
“Parallel Access to Files in the Vesta File System”, in Proc.
Supercomputing ’93, pp. 472-481, November 1993.
[3] Crandall, P. E., Aydt, R. A., Chien A. A., and Reed, D. A.,
“Input/Output Characterization of Scalable Parallel Applications”, in Proc. Supercomputing ’95, December 1995.
[4] Henderson, R. D., and Karniadakis G. E., “Unstructured
Spectral Element Methods for Simulation of Turbulent
Flows", J. Comput. Phys., Vol. 122(2), pp. 191-217, 1995.
[5] Henderson, R. D., Private Communication, January 1996.
[6] Huber, J., Elford, C. L., Reed, D. A., Chien, A., and Blumenthal, D. S., “PPFS: A High Performance Portable Parallel
File System”, in Proc. 9th ACM International Conference on
Supercomputing, pp. 385-394, July 1995.
[7] Kotz, D. and Nieuwejaar, N., “Dynamic File-Access Characteristics of a Production Parallel Scientific Workload”, in
Proc. Supercomputing ’94, pp. 640-649, November 1994.
[8] LoVerso, S. J., Isman, M., Nanopoulos, A., Nesheim, W.,
Milne, E. D., and Wheeler, R., “sfs: A Parallel File System
for the CM-5", in Proc. 1993 Summer USENIX Conference,
pp. 291-305, 1993.
[9] Miller, E. L., and Katz, R. H., “Input/Output Behavior of Supercomputing Applications”, in Proc. Supercomputing ’91,
pp. 388-397, November 1991.
[10] Nieuwejaar, N., Kotz, D., Purakayastha, A., Ellis, C. S., and
Best, M., “ File-Access Characteristics of Parallel Scientific
Workloads", PCS-TR95-263, Dept. of Computer Science,
Dartmouth College, August 1995.
[11] Nitzberg, B., “Performance of the iPSC/860 Concurrent
File System”, RND-92-020, NAS Systems Division, NASA
Ames, December 1992.
[12] Pasquale, B. K., amd Polyzos, G., “A Static Analysis of
I/O Characteristics of Scientific Applications in a Production Workload”, in Proc. Supercomputing ’93, pp. 388-397,
November 1993.
[13] Pasquale, B. K., amd Polyzos, G., “Dynamic I/O Characterization of I/O Intensive Scientific Applications”, in Proc.
Supercomputing ’94, pp. 660-669, November 1994.
[14] Poole, J. T., “Preliminary Survey of I/O Intensive Applications", California Institute of Technology,
http://www.ccsd.caltech.edu/SIO/SIO.html, 1994.
[15] Purakayastha, A., Ellis, C. S., Kotz, D., Nieuwejaar, N.,
and Best, M., “Characterizing Parallel File-Access Patterns
on a Large-Scale Multiprocessor”, in Proc. 9 th International
Parallel Processing Symposium, pp. 165-172, April 1995.
[16] Reed, D. A., Aydt, R. A., Noe, R. J., Roth, P. C., Shields,
K. A., Schwartz, B. W., and Tavera, L. F., “Scalable Performance Analysis: The Pablo Performance Analysis Environment”, in Proc. Scalable Parallel Libraries Conference, A.
Skjellum (ed.), IEEE Computer Society, pp. 104-113, 1993.
[17] Reed, D. A., Elford C.L., Madhyastha T., Scullin W.H.,
Aydt, R. A., and Smirni E., “I/O, Performance Analysis, and
Performance Data Immersion”, in Proc. MASCOTS’96, pp.
5-16, 1996.
[18] Winstead, C., Pritchard, H., and McKoy, V., “Parallel Computation of Electron-Molecule Collisions”, IEEE Computational Science & Engineering, pp. 34-42, 1995.
[19] Winstead, C., Private Communication, January 1996.