Submitted to the 4th European Dependable Computing Conference (EDCC4), Toulouse, France - October 23-25, 2002
Towards a Framework for Dependability Benchmarking
H. Madeira*, K. Kanoun**, J. Arlat**, D. Costa***, Y. Crouzet**, M. Dal Cin+, P. Gil++ and N. Suri+++
*
+
**
***
University of Coimbra, Portugal
LAAS-CNRS, Toulouse, France
Critical Software, Coimbra, Portugal
++
+++
FAU, Erlangen-Nürnberg, Germany
UPV, Valencia, Spain
Chalmers University, Göteborg, Sweden
Abstract
The goal of dependability benchmarking is to provide generic ways for characterizing the
behavior of components and computer systems in the presence of faults, allowing for the
quantification of dependability measures. Beyond existing evaluation techniques, dependability
benchmarking must provide a reproducible and cost-effective way of performing this evaluation
either as stand alone assessment or more often for comparative evaluation across systems. This
paper proposes a framework for defining dependability benchmarks for computer systems, with
particular emphasis on COTS and COTS-based systems. The multiple dimensions of the
problem are discussed and the framework is presented through a set of dependability
benchmarking scenarios, ranging from experimental approaches to benchmarking scenarios
where modeling and experimentation are tightly coupled. The main problems and proposals
behind each benchmarking scenario are discussed and concrete case studies presented.
Keywords:
Dependability
Characterization,
Dependability
Assessment,
COTS components and COTS-based Systems.
Submission category: Regular paper
Word count: 6500
The material included in this paper has been cleared through authors’ affiliations
Contact author:
Henrique Madeira
DEI-FCTUC
Polo II, University of Coimbra
3030 Coimbra
Portugal
Phone: (+351) 239 790003
Fax: (+351) 239 790003
Email: henrique@dei.uc.pt
Benchmarking,
1 Introduction
The goal of benchmarking the dependability of computer systems is to provide generic ways for
characterizing their behavior in the presence of faults. Benchmarking must provide a reproducible way for
dependability characterization. The key aspect that distinguishes benchmarking from existing evaluation and
validation techniques is that a benchmark fundamentally should represent an agreement that is accepted by
the computer industry and by the user community. This technical agreement should state the measures, the
way and conditions under which these measures are obtained, and the validity domain. A benchmark must
be as representative as possible of a domain. The objective is to provide practical ways to characterize the
dependability of computers, so as to help system manufacturers and integrators improving their products as
well as end-users in their purchase decisions.
The success of well-established performance benchmarks in comparing performance of components and
computer systems probably accounts for the generalized idea that the main goal of benchmarks is to compare
systems on the basis of benchmark results. However, we would like to emphasize that dependability
benchmarks can be used in a variety of cases by both the end-users and manufacturers:
•
Characterize the dependability of a component or a system.
•
Identify weak parts of a system, requiring more attention and perhaps needing some improvements by
tuning a component to enhance its dependability (e.g., using software wrappers), or by tuning the system
architecture (e.g., adding fault tolerance) to ensure a suitable dependability level.
•
Compare the dependability of alternative or competitive solutions according to one or several
dependability attributes.
This paper proposes a framework for defining dependability benchmarks for computer systems, with
emphasis on Commercial-Off-The-Shelf (COTS), and COTS-based systems, via experimentation and
modeling.
Although a variety of evaluation techniques exist, aimed at different system levels and domains, a
generalized approach that can be used to evaluate and compare different systems and applications still does
not exist. Instead, several methods have been used, ranging from analytical modeling to simulation and
experimental approaches, including those based on fault injection, e.g., see [1-7]. These approaches form a
natural basis for our proposal for dependability benchmarking and the following paragraphs refer to those
that are most appropriately related to dependability benchmarking.
It is worth mentioning the pioneering work that addressed the benchmarking of software robustness [8] and
investigated the development of some forms of benchmarks [9, 10]. Among these studies, operating systems
have received much attention, e.g., see [11-13] including development of tools (respectively, Ballista, Fuzz
and MAFALDA) to support robustness testing. On the other hand, performance benchmarking is now a
well-established area that is led by organizations such as the TPC (Transaction Processing Performance
2
Council) and SPEC (Standard Performance Evaluation Corporation), and supported by major companies in
the computer industry, e.g., see [14].
Clearly, the preliminary efforts at developing dependability benchmarks, including focused robustness
testing techniques, do not benefit yet from such a level of recognition. Much work is still needed despite the
many relevant efforts that have tackled in a unified way several issues concerning performance and
dependability assessment. Among these, we would like to point out the following studies that addressed
jointly a couple of such issues: bringing together performance benchmarking and dependability assessment
[15], field measurement and fault injection [16], field measurement and modeling [17], fault injection and
modeling [18], and use of standard performance benchmarks as a workload for dependability evaluation
[19]. To the best of our knowledge, little has been reported to cover these various facets in a comprehensive
manner. Accordingly, this is the path that is specifically being explored in our work. This paper elaborates
on our first thoughts reported in [20].
The ultimate objective of our work is to comprehensively define dependability benchmarks for computer
systems, and provide means for implementing them. In this paper, we identify the main dimensions that are
decisive for defining dependability benchmarks and show how they can be integrated into benchmarking
scenarios to assess benchmark measures. These dimensions allow the description of i) the considered system
in its operating environment, ii) the benchmark context, iii) the benchmark measures and iv) all what is
needed for experimentation. A benchmark is defined when all of these dimensions are specified. Three
examples of case studies are provided to exemplify how the dimensions of the proposed framework can be
instantiated in real situations. The first one addresses operating systems that are the core components in
computer systems. The other examples focus on transactional and embedded systems and applications.
It is expected that these case studies will aid defining the dependability benchmarks, at least for the
considered classes of systems.
The paper is structured into five sections. Section 2 presents the relevant dimensions that characterize
dependability benchmarking. Section 3 illustrates how the framework can be used for conducting
dependability benchmarks. Section 4 describes the main specifications of preliminary benchmarks for three
case studies. Finally, Section 5 concludes the paper.
2
Dependability Benchmarking Dimensions
The definition of a meaningful framework for dependability benchmarking requires first of all a clear
understanding of all impacting dimensions. Their examination is essential to appreciate the problem space
and to classify and specify all aspects of dependability benchmarks. Figure 1 outlines our classification of
dimensions. The categorization dimensions allow us to organize the dependability benchmark space into
well-identified categories. These dimensions describe the considered system and the benchmarking context.
The measure dimensions specify the dependability benchmarking measure(s) to be assessed depending on
3
the choices made for the categorization dimensions. The experimentation dimensions include all aspects
related to experimentation on the target system to get the base data needed to obtain the selected measure(s).
Categorization
Measure
Considered system
(system nature
application area
operating environment)
Benchmarking context
(Life-cycle phase
benchmark user
benchmark purpose
result scope
benchmark performer)
Measure nature
(qualitative / quantitative)
Experimentation
Workload
Faultload
Measure type
(dependability / performance-related)
Measurements
Measure extent
(comprehensive / specific)
Assessment method
(experimentation / modeling)
Figure 1 - Dependability benchmarking dimensions
In addition to these dimensions, a benchmark must fulfill a set of properties to be considered valid and
useful. For example, it must be reproducible (in statistical terms), must not cause undesirable interferences
or perturbations on the target system, must be portable, cost-effective, etc. These properties represent goals
that must be achieved when defining actual dependability benchmarks.
In the following sub-sections we concentrate on detailing the three groups of dimensions as depicted in
Figure1.
2.1
Categorization Dimensions
The categorization dimensions impact the basic selection of meaningful benchmark measures as well as the
experimentation dimensions.
2.1.1
Considered System
It is essential to clearly identify the system under consideration, namely, its nature, its application area, as
well as its operating environment. We detail each of them below.
System Nature - The system ranges from discrete hardware and software components (e.g., operating
systems, middleware) to complete systems. Two conflicting views determine the way the system should be
described in a dependability benchmark. On one hand, it would be desirable to know the system in detail to
make possible the evaluation of specific measures such as measures directly related to fault tolerance
mechanisms available in the system. On the other hand, a detailed description of the system could drastically
reduce the portability of a benchmark. As a compromise, we propose two complementary views:
•
Abstraction layer view: The target system is described in a generic way as a set of abstraction layers (or
components) that perform specific functions, including key components for dependability. The main
4
question is how to reconcile the need of describing the target system to a given detail with the necessary
level of portability required for a benchmark.
•
Functional view: The system is described in terms of functionalities and features, including dependability
features. The main question to be investigated is the extent to which this approach affects the accuracy of
the various measures that can be obtained from the benchmarks. One possible way could be to associate,
to some measures, disclosures specifying the detailed knowledge of the system required to obtain that
measure.
Application Area - The application area is a key dimension for the definition of dependability benchmarks,
as it impacts the system activation profile, the operating environment and the selection of benchmark
measures. The division of the application spectrum into well-defined application areas is necessary to cope
with the diversity of applications. The main difficulty is clearly the establishment of the appropriate
granularity to divide the application spectrum. Obviously, different application areas will need different
dependability benchmarks. For example, telecommunication applications have requirements that are
different from those for control applications. The case studies presented in Section 4 illustrate this point.
Operating Environment - The operating environment characterizes the environment in which the system is
used in operation. It includes aspects such as maintenance actions. As a consequence, it may affect the
workload in a way that is usually ignored in performance workloads that concentrate essentially on
functional aspects. Additionally many computer faults are induced by external sources that are intimately
related to the operating environment. The main problem is to identify a way to take into account this
environment. This is even more difficult when including human aspects. However, if the application area is
well defined and bounded, it should be possible to define a typical operating environment to be used in
experiments, including the set of operator faults that are agreed upon as typical for that application area.
2.1.2
Benchmarking Context
Performing a dependability benchmark and using the benchmark results can be done with multiple
perspectives. The benchmarking context is, in fact, a composite dimension including the following:
Life Cycle Phase - A dependability benchmark could be performed in any phase of the system life cycle. The
results could be used for the current or subsequent phases.
Benchmark user - Person or entity actually using the benchmark results.
Benchmark purpose - Results can be used either to characterize system dependability capabilities in a
qualitative manner, to assess quantitatively these capabilities, to identify weak points or to compare
alternative systems. The purpose may vary along the life-cycle, for example, during development, results
could be used as a validation means or for identifying weak parts of the system and in operational life, they
could be used to evaluate the impact of physical and design faults on system dependability.
5
Results scope - Benchmark results can be used internally or externally. External use involves standard results
that fully comply with the benchmark specification, for public distribution, while internal use is mainly for
system validation and tuning.
Benchmark performer - Person or entity performing the benchmark (e.g., manufacturer, integrator, third
party, end-user). These entities have i) different visions of the target system, ii) distinct accessibility and
observability levels for experimentation and iii) different expectations from the measures.
2.2
Measure Dimensions
Dependability benchmarks encompass measures assessed under particular conditions. Therefore an
important task in benchmark definition concerns the identification of meaningful measures. Such measures
allow the characterization of the system in a qualitative manner (in terms of features related to the system
dependability capabilities and properties) or quantitatively.
As the occurrence or activation of faults may lead to performance degradation without leading to system
failure, dependability and performance are strongly related. Thus the evaluation of system performance
under faulty conditions (referred to as performance-related measures) allows completely characterizing the
system behavior from a dependability perspective.
The various context perspectives impact the type of benchmark measures. Typically, end-users are interested
in measures defined with respect to the expected services, referred to as comprehensive measures, while
manufacturers and integrators could be more interested in specific measures.
Comprehensive measures characterize the system at the service delivery level, taking into account all events
impacting its behavior. They may require the use of both experimentation and modeling. It is worth
mentioning that this constitutes a new aspect in the context of computer benchmarking, as performance
benchmarks rely only on direct measures.
A specific measure is usually associated with a specific system feature without necessarily taking into
account all processes impacting its behavior (e.g., failure rates, maintenance duration). It may concern, for
instance, error detection and recovery or fault diagnosis capabilities. For example, two specific measures can
be associated to error detection: i) coverage of error detection mechanisms and ii) error latency (time
required to detect an error). Such measures can be evaluated based on experimentation.
The actual set of benchmark measures is very dependent on the categorization dimensions. Moreover, more
than one measure can be evaluated for a given set of categorization dimensions. Indeed a benchmark
"measure" can be considered as a vector of measures, some of them being qualitative features, and some
being quantitative measures.
6
2.3
Experimentation Dimensions
These include all aspects related to experimentation on the target system to obtain the selected measures, i.e.,
the workload, the faultload and measurements to be performed on the target system. Thus, these three
dimensions are defined according to the categorization and measure dimensions.
2.3.1
Workload
The workload represents a typical operational profile for the considered application area. Widely accepted
performance benchmarks can provide workloads for many application areas and clearly show that it is
possible to agree on an abstract workload that reasonably represents a given application area. However, in
dependability benchmarking, the concept of workload must be expanded to take into account the impact of
preventive and corrective maintenance actions that are always conducted as part of routine operations.
2.3.2
Faultload
The faultload consists of a set of faults and exceptional conditions that are intended to emulate the real
threats the system would experience. Internal faults (hardware or software faults) are mainly determined by
the actual target system implementation. Considering an approach in which few assumptions are made on
the actual target system structure (physical and logical), it is likely that the internal faults should be
represented in the faultload by high-level classes of faults. In this context, equivalent sets of faults for
different target systems must be understood on a statistical basis (used in the same application area) rather
than being literally identical. External faults clearly depend on the operating environment.
The definition of the faultload will inevitably be a pragmatic process based on knowledge, observation, and
reasoning. Inputs from many different application areas are needed to accomplish this task with success.
Some examples are information from faults in the field (see e.g., [21] for software faults), characteristics of
the operating environment (for operator faults), or even information from experimental and simulation
studies.
The definition of representative faultloads is probably the most difficult part of defining a dependability
benchmark. To make this problem tractable we are focusing, as a start, on well-defined application areas and
considering faultloads that address only one class of faults at a time.
2.3.3
Measurements
Measurements performed on the target system allow the observation of the reaction of the system to the
applied workload and faultload. The measures of interest are then obtained from processing these
measurements.
7
2.4
Summary
This section allowed us to identify all dimensions required to define a dependability benchmark. The
categorization dimensions define clearly the considered system (including its operating environment) and the
benchmark context. Based on these dimensions, one or more benchmark measure(s) of interest can be
defined. From an experimental point of view, the categorization and measure dimensions impact the three
experimentation dimensions: the workload, the faultload and the measurements to be performed on the target
system. Thus a central problem concerns the definition of suitable experimentation dimensions taking into
account the other dimensions in an appropriate way.
3
Benchmark Conduct
A fundamental question entails ascertaining the final form of the product “dependability benchmark”? At
this stage, giving the large spectrum of dimensions to take into account, we think that it is not reasonable to
expect to be able to provide dependability benchmarks in a form that is “ready to use”, integrating all
dimension choices consistently. As a result, the shape of dependability benchmarks could be a detailed
specification including a list of procedures and rules to guide all the processes of producing benchmark
measures, from experimentation and modeling. In particular, a dependability benchmark should include
procedures and rules defining the way the specification should be implemented in the target system as well
as procedures and rules for conducting experiments and to ensure uniform conditions for measurement.
We are working towards these objectives and we expect that the case studies considered in Section 4 will
help us to define such specifications, at least for the considered classes of systems.
Assuming that such specifications do exist at least for some classes of systems, we now discuss the various
steps required for conducting the dependability benchmark of a particular system or component.
Benchmarking is achieved in several steps forming a benchmarking scenario. Benchmarking starts by an
analysis step that is necessary to settle on all dimensions (in many cases this step can be very simple).
Characterization and assessment of the target system dependability is based on an experimentation step
according to the selected dimensions. If comprehensive measures of the target system are of interest, a
modeling step is required in addition to experimentation. Examples of modeling techniques for evaluating
comprehensive measures are: block diagrams, faults trees, Markov chains or stochastic Petri nets.
Figure 2 gives the three steps for dependability benchmarking and their interrelations (links A to E). To
illustrate how this general benchmarking framework can be used in real situations, we have chosen four
examples of scenarios ranging from a purely experimental one to a more complete one in which modeling
and experimentation are tightly coupled.
The rest of this section presents the three benchmarking steps and their interrelations, and introduces the four
examples of benchmarking scenarios before discussing some related research issues.
8
Analysis
B
Modeling
Comprehensive
Measures
A
C
D
E
Workload
Faultload
Target System
Experimental
Measures
& Features
Experimentation
Figure 2 - Dependability benchmarking steps
3.1
Benchmarking Steps
The analysis step consists in defining the categorization and measure dimensions for the target system. If a
benchmark corresponding to these dimensions is already available, the experimentation dimensions can be
derived directly from this benchmark. Otherwise, they should be specified.
Two kinds of outputs are expected from the analysis depending on the benchmark measure assessment
method. For experimental measures, outputs A in Figure 2 give the selected workload, faultload and
measurements.
If modeling is required, the analysis step may, in addition, consist in analyzing deeply the system behavior to
prepare modeling (outputs B). In which case, the selection of the workload, faultload and measurements can
be guided by the modeling step as well.
The experimentation step pertains to carrying out the experimental part of the benchmark, i.e., executing
the workload and faultload and collecting the response of the system as a result of measurements. The
experimental measures are then obtained from the collected data.
However, as experimentation is tightly related to the target system, the workload and faultload defined at a
high level during the analysis step or by the already existing benchmark specification, should be transformed
or refined in order to be implemented at the target system level. Also, measurements are tightly related to the
target system accessibility and observability points. Indeed, the benchmark should specify the kind of
outputs required from the experiments to assess the benchmark measures. Deep examination of the target
system allows determination of the accessibility and observability required for obtaining the specified
outputs.
This refinement should be accomplished according to the procedures and rules included in the benchmark. In
particular, these procedures and rules should address: scalability of the benchmark, configuration
disclosures. Also, such rule should avoid "gaming", to produce biased results
Finally, it is worth mentioning that when the benchmark purpose is to identify system capabilities and
possible weaknesses, no numerical values are to be estimated.
9
The modeling step lies in the description of the system behavior as well as interactions between
components, taking into account failure and repair rates of system components. Processing of the analytical
model to assess the measures requires the allocation of numerical values to the model parameters. Some of
such values may be obtained from experimentation on the target system (e.g., coverage factors, restart time).
Other parameter values can be provided from field data or based on past experience related to similar
systems (e.g., failure and repair rates).
It is worth mentioning that models of large systems made of several components with several dependencies
may be very complex (see e.g., [22]). However, for some systems, mainly COTS-based systems for which
the details about the system architecture are not known, simple high-level models can be constructed. Such
models may require only a reduced effort, thus offering a good trade-off for obtaining comprehensive
measures characterizing globally the system with an acceptable effort.
Usually experimentation and modeling are used in a complementary way to assess system dependability
[23]. Examples of relationships between modeling and experimentation are shown in Figure 2:
•
Link C: Modeling is used to guide the selection of the most relevant classes of workload and faultload.
•
Link D: Modeling is used to guide the selection of experimental measures and features. Actually, the
constructed model is processed and sensitivity analyses with respect to parameter values are achieved to
identify the most significant ones (i.e., associated to the most salient features of the system) that need to
be accurately assessed by experimentation.
•
Link E: The experimental results may lead to the refinement/correction of the analytical model. For
example, the model may assume two failure modes, if experimentation shows the existence of a third one
with a significant probability, the original model should be updated.
3.2
Benchmarking Scenarios
We have selected four examples of scenarios. The complexity of benchmarking and the associated effort
increase from the simplest scenario (S1), based only on experimentation, to the most complete one (S4), in
which modeling and experimentation are tightly coupled. However, more results can be obtained from the
most complete one. Indeed, S1 is included in the three other scenarios. Hence, all the problems and research
issues for S1 are also relevant for the others. These problems and research issues are briefly presented in
Section 3.3.
S1: Dependability Benchmarking Based Only on Experimentation - The simplest scenario for
benchmarking dependability follows a pure experimental approach. It includes analysis and experimentation
steps and link A of Figure 2. It is actually an extension of the well-established performance benchmark
setting. The analysis step specifies a set of measures and all the experiments necessary to obtain them.
Typically, the target system is described in a functional way and few assumptions are made regarding its
internal architecture. This approach can be used even when the documentation is not completely available as
for COTS and COTS-based systems.
10
S2: Experimentation Supported by Modeling - S2 differs from S1 by the fact that experimentation is
guided, at least partially, by modeling. It includes the three steps and links A, B, C and D. Modeling helps in
selecting features and measures to be assessed experimentally. The outputs are similar to those of S1:
experimental measures and features.
S3: Modeling Supported by Experimentation - S3 includes the three steps and links A, B and E. In S3,
some specific experimental measures, used in the analytical model representing the system behavior, may be
provided by experimentation. Moreover, system features identified by experimentation may impact the
model semantics. Therefore, experimentation supports model validation and refinement. The outputs are
comprehensive measures obtained from modeling. However, the experimental measures and features,
assessed for supporting modeling, may be made available.
S4: Modeling and Experimentation S4 is a combination of S2 and S3. In this scenario, not only the
selection of measures and features to be assessed experimentally are guided, at least partially, by modeling,
but modeling is also supported by experimental results. Its outputs are experimental measures and features,
as well as comprehensive measures based on modeling.
3.3
Some Research Issues
Representativeness is a crucial concern, as benchmark results must characterize the addressed aspects of the
target system in a realistic way. Regarding established performance benchmarks, the problem is reduced to
the representativeness of performance measures and of the workload. For dependability, it is also necessary
to define representative dependability measures and representative faultloads. Although the problem seems
clearly more complex than for performance benchmarks, the pragmatic approach used in the established
performance benchmarks offers a basis for identifying adequate solutions for dependability benchmarking
representativeness.
It is worth mentioning that many technical problems still need to be resolved. The subsequent points
summarize crucial research issues.
•
The adoption of the workloads of established performance benchmarks is the starting point for the
definition of workloads. However, some work still has to be done. For example one has to check whether
the way the application spectrum as partitioned by the performance benchmarks is adequate for this new
class of dependability benchmarks.
•
The definition of representative faultloads encompasses specific problems that are currently being studied
(such as fault representativeness and the definition of practical way to apply the faultload to the system).
•
Definition of meaningful measures. In particular, special attention should be paid to confidence and
accuracy of measurements and the possible impact of the measurements on system behavior
(intrusiveness).
11
4 Case Studies
In this section, we illustrate how the general concepts presented in the previous sections can be applied to
examples of preliminary benchmarks. To study a representative set of benchmarks, thus allowing for the
investigation of a substantial range of benchmark dimensions, we are considering three categories of target
systems: i) general purpose operating systems, ii) transactional systems and iii) embedded systems.
Concerning embedded systems, two classes of application domains are considered (space and automotive),
whose implementations are respectively more software-oriented and hardware-oriented. In particular, the
automotive application features a system-on-chip implementation.
To facilitate the cross-exploitation of the results and insights obtained when carrying out these studies, we
chose to consider as much as possible a set of common benchmark categorization dimensions. Accordingly,
hereafter, after describing the benchmark categorization dimensions, the emphasis will be put on the
presentation of the most salient experimentation dimensions as well as of the measures.
4.1 Categorization dimensions
These dimensions are common to all case studies and are defined as follows:
•
Target system: The target system is always either an Off-the-Shelf component, either commercial (COTS)
or Open System Software (OSS) or a system including at least one such component.
•
Life cycle phase: The benchmark is performed during the integration phases.
•
Benchmark user: The primary users are the integrators of the target system. However, the benchmark
results are to be communicated to the successive levels of integrators up to the end-users of the
component or of the integrated system. Nevertheless, some results may be of interest to the developer(s)
of the benchmarked OTS component, for improving the concerned component, in case the benchmark
reveals some deficiencies.
•
Benchmark purpose: For all target systems, the following possible purposes are identified: i) assess some
dependability features, ii) assess dependability (and performance) related measures, and iii) identify
potential weaknesses.
•
Result scope: The benchmarks considered are primarily meant for external use (i.e., performed in a
standardized way and made publicly available).
•
Benchmark performer: The benchmark performer is either the system integrator or a third party. Indeed,
we consider that the benchmark performer is someone (or an entity) who has no in depth knowledge
about the component(s) being integrated and who is aiming at i) improving significantly its knowledge
about the related dependability features, and ii) publicizing information on the dependability of the
component and of the system under integration in a standardized way.
12
4.2 General Purpose Operating Systems
Operating Systems (OSs) form a generic software layer that provides basic services to the applications
through an application-programming interface (API). Thus, they form privileged target software components
for conducting dependability characterizations [24].
Two main approaches can be identified for benchmarking OS dependability, as viewed by the basic services
a general purpose OS typically provide. First, a detailed analysis of candidate OSs can be carried out by
decomposing them according to the set of common services they provide. Such decomposition can be
mapped to a set of generic functional components that constitute the OS (e.g., synchronization, scheduling,
memory management, driver controls and I/O functionality). It is very much suited for kernel components
(especially for OSs based on microkernels (e.g., see [13]), but can be applied to a large extent to general
purpose OSs, especially OSSs. An alternative way to perform a fair analysis can be obtained when the OSs
exhibit a similar API. A typical example of such a popular API is the POSIX interface (e.g., see [11]).
The benchmark prototypes under investigation implement the dependability benchmark for general purpose
OS in two well-known OSs: Linux and Windows 2000. They differ significantly in their internal architecture
and in their implementation, which constitutes an excellent test of the benchmark portability. Additionally,
the two approaches for OS benchmarking mentioned above are being investigated in these prototypes.
Workloads and Faultloads - As workload, we consider synthetic workloads respectively focused to the
main functional components that have been identified above. These workloads correspond to specific
programs aimed at activating the services provided by the functional components. For the faultloads, based
on previous studies [25, 26], and as we are targeting COTS OSs, for which data on internal structure is
usually not available, we concentrate on failure of the applications when interacting with the OS, by
providing both invalid and valid erroneous parameters to the target OS.
Measures - At OS-level, we focus on the failure modes exhibited and that are characterized by the following
events: i) kernel hang or crash, ii) exception and iii) error status report. Hindering (the error code returned is
not correct) is also an interesting facet of the behavior [11]. It is important to note that these outcomes are
not exclusive (several of such events can occur as the result of the application of a specific faultload). Thus,
related timing observations are also of interest, as the order of occurrence of these events is important (e.g.,
the observation of an exception before a kernel hang is usually preferred to the opposite). Although precise
timing measurements are not easy to achieve, especially in the case of a COTS OS (except for the case of
exceptions), it might only be possible to observe the ordering of events, in this case.
It is worth noting that OS Reboot/Reset times resulting from the events listed above are also of interest. OSspecific timing measurements (such as switching time) should be more relevant for control applications than
for transactional applications. Accordingly, it is worth pointing out that these specific analyses will be
complemented by the application-specific benchmarks, as they also form the underlying OSs for the
transactional and embedded systems considered in the next two sections.
13
4.3 Transactional Systems
We specifically address OLTP (On-Line Transaction Processing) systems, as they constitute the kernel of
the information systems used to support the daily operations in most of businesses (banking, insurance,
travel, health care, etc.). Such applications form traditional examples of business-critical areas, which clearly
justify choosing them as targets to devise a dependability benchmark for transactional applications. The
target system uses the Oracle database running on Windows 2000.
Workloads and Faultloads - We adopted the workload of the well-established TPC Benchmark™ C (TPCC) [27]. It represents a mix of read-only and update intensive transactions that simulates the activities of
most common OLTP environments, including transactions resulting from human operators working on
interactive sessions. Concerning the faultloads, besides software faults (that can be emulated to some extent
by machine-code level errors [28]), we emphasize also the analysis of the target system subjected to faults
caused by human operators interacting with the system.
Measures - We naturally favor service-oriented measures that reflect the end-user point of view. Both
dependability- and performance-related measures are being considered. The performance-related measure
considered is the number of transactions executed per unit of time in the presence of a (standard) faultload. It
measures the impact of faults on the performance. It favors systems with higher capability of tolerating
faults, fast recovery time, etc. As is the case for performance benchmark, it might be useful to complement
this measure with a cost-related measure (e.g., the price of processing transactions in the presence of a
(standard) faultload. The main interest is to account for the benefit of including fault tolerance mechanisms
in the system. It is worth noting that what matters here are not absolute performance figures, but rather the
relative measures (e.g., degradation ratio due to the faultload).
In addition to such measures, it is possible to extend the observation to the behavior, in presence of faults, of
particular system components. For example, the occurrence or activation of a fault may trigger the error
detection mechanisms in the target system at various levels: hardware, OS, database engine and application.
The fact that in the benchmark prototypes under investigation the transactions are processed on top of the
same operating systems (i.e., Windows 2000 and Linux) used in the experiments with the benchmarks for
general purpose OSs will provide further insights on the behavior of the OS in presence of faults. Other
specific measures concern error detection and/or recovery latency and coverage. Finally, violations of
properties (e.g., the well-known ACID properties: Atomicity Consistency, Isolation and Durability) provide
another set of specific measures.
4.4 Embedded Systems
The two considered target systems are control applications from space and automotive industries. They both
run on top of Linux. The study of two such target systems is meant to better account for the spectrum of
work that could be performed for embedded systems. The first (software-oriented) system is a
control/navigation system for satellite applications. The software was originally written in Ada language and
14
is based on the special Thor processor developed by Saab Ericsson Space [29]. It is being used in the Odin
satellite that is currently operational. The system has been ported to the C language, using the POSIX
standard. The second system features a system on-a-chip architecture for automotive application. The target
system is a microcontroller (MPC565). The application is a subset of an electronic control unit for a Diesel
engine. Two distinct architectures are being studied: a standard one featuring a single microcontroller, and a
duplex architecture composed of two microcontrollers interconnected by a CAN network.
Workloads and Faultloads - The workloads are very much dependent of the applications running on each
target system and thus will be tailored to these applications. For the space application, the faultload will be
focused to test the robustness of the underlying OS at the level of the POSIX-compliant API. In the case of
the system-on-a-chip target system, more emphasis will be put on simulating by means of bit-flips the
consequences of hardware faults using the software implemented fault injection technique. The distribution
and multiplicity of injected bit-flips will be based on results obtained from preliminary fault injection studies
conducted using a VHDL simulator [30].
Measures - Here again, the main measures of interest concern the services offered by the respective
applications. As both applications are real-time control applications, we are concerned with failures in the
value domain (erroneous results) as well as on timing failures (such as deadline misses). This will allow us
to extend the observations of the behavior of the OS in presence of faults, in particular to account for
specific timing measurements. The latter include system call overhead, context-switch, etc. It is important to
mention that the embedded applications context also needs to be considered to define the relevant measures
accordingly. For the considered control applications, the additional (non-traditional) measures include
aspects such as jitter, lag and stability that have ramification on the delivery of the services expected from
the application. In addition, emphasis will be put on measures dedicated to assess the fault tolerance
mechanisms featured by the systems, including hardware redundancy and exception handling mechanisms.
Finally, the fact that both applications run on Linux will contribute to enhance the insights already obtained
for this target OS. In particular, building upon the framework proposed in [7], observation means will be
enhanced to allow for a detailed analysis of error propagation channels, mainly for internal purpose.
5 Conclusion
This paper was devoted to dependability benchmarks for computer systems, with particular emphasis on
COTS components and COTS-based systems. Our goal is to determine generic ways for defining
dependability benchmarks. Given the multi-dimensional nature of the problem, we started by the
identification of all the dimensions of the problem. Three groups of dimensions have been identified:
categorization, measure and experimentation dimensions.
The framework proposed for conducting dependability benchmarks encompasses three main steps.
Benchmarking starts by an analysis step that is necessary to enable the choice of all dimensions.
Characterization and assessment of the target system dependability is based on an experimentation step
15
according to the selected dimensions. If comprehensive measures of the target system are of interest, a
modeling step is required in addition to experimentation. To illustrate how this framework can be used in
real situations, we proposed four examples of benchmarking scenarios ranging from a purely experimental
one to a more complete one in which modeling and experimentation are tightly coupled.
Finally, examples of dependability benchmarks concerning operating systems, transactional and embedded
systems were presented. These benchmarks are still under study in the DBench project [31]. In particular,
experiments will be set up to check the conceptual framework. More generally, a lot of work is still needed
to validate the concepts introduced in this paper and to define dependability benchmarks in a comprehensive
way. Due to the large spectrum of dependability benchmarking, we will concentrate our work essentially on
the classes of target systems presented in this paper and expect to be able to define generic benchmarks at
least for these classes of systems.
References
[1]
M. Malhotra and K. S. Trivedi, “Dependability Modeling Using Petri Nets”, IEEE Transactions on
Reliability, vol. 44, no. 3, pp. 428-440, September 1995.
[2] J. Karlsson, P. Folkesson, J. Arlat, Y. Crouzet, G. Leber and J. Reisinger, “Application of Three
Physical Fault Injection Techniques to the Experimental Assessment of the MARS Architecture”, in
Dependable Computing for Critical Applications (Proc. 5th IFIP Working Conf. on Dependable
Computing for Critical Applications: DCCA-5, Urbana-Champaign, IL, USA, September 1995), (R.
K. Iyer, M. Morganti, W. K. Fuchs and V. Gligor, Eds.), pp. 267-287, Los Vaqueros, CA, USA: IEEE
Computer Society Press, 1998.
[3] J. V. Carreira, D. Costa and J. G. Silva, “Fault Injection Spot-checks Computer System
Dependability”, IEEE Spectrum, vol. 36, 50-55, August 1999.
[4] R. J. Martínez, P. J. Gil, G. Martín, C. Pérez and J. J. Serrano, “Experimental Validation of HighSpeed Fault-Tolerant Systems Using Physical Fault Injection”, in Dependable Computing for Critical
Applications (Proc. 7th IFIP Working Conf. on Dependable Computing for Critical Applications:
DCCA-7, San Jose, CA, USA, January 1999), (C. B. Weinstock and J. Rushby, Eds.), pp. 249-265,
San Jose, CA, USA: IEEE Computer Society Press, 1999.
[5] M. Dal Cin, G. Huszerl and K. Kosmidis, “Quantitative Evaluation of Dependability-Critical Systems
Based on Guarded Statechart Models”, in Proc. 4th Int. High-Assurance Systems Engineering Symp.
(HASE-99), Washington, DC, USA, 1999, pp. 37-45 (IEEE CS Press).
[6] J. Arlat, J. Boué and Y. Crouzet, “Validation-based Development of Dependable Systems”, IEEE
Micro, vol. 19, no. 4, pp. 66-79, July-August 1999.
[7] M. Hiller, A. Jhumka and N. Suri, “An Approach for Analysing the Propagation of Data Errors in
Software”, in Proc. Int. Conference on Dependable Systems and Networks (DSN-2001), Göteborg,
Sweden, 2001, pp. 161-170 (IEEE CS Press).
[8] A. Mukherjee and D. P. Siewiorek, “Measuring Software Dependability by Robustness
Benchmarking”, IEEE Transactions of Software Engineering, vol. 23, no. 6, pp., 1997.
[9] T. K. Tsai, R. K. Iyer and D. Jewitt, “An Approach Towards Benchmarking of Fault-Tolerant
Commercial Systems”, in Proc. 26th Int. Symp. on Fault-Tolerant Computing (FTCS-26), Sendai,
Japan, 1996, pp. 314-323 (IEEE Computer Society Press).
[10] A. Brown and D. A. Patterson, “Towards Availability Benchmarks: A Cases Study of Software RAID
Systems”, in Proc. 2000 USENIX Annual Technical Conference, San Diego, CA, USA, 2000
(USENIX Association).
[11] P. Koopman and J. DeVale, “Comparing the Robustness of POSIX Operating Systems”, in Proc. 29th
Int. Symp. on Fault-Tolerant Computing (FTCS-29), Madison, WI, USA, 1999, pp. 30-37 (IEEE CS
Press).
[12] J. E. Forrester and B. P. Miller, “An Empirical Study of the Robustness of Windows NT Applications
Using Random Testing”, in Proc. 4th USENIX Windows System Symposium, Seattle, WA, USA, 2000.
16
[13] J. Arlat, J.-C. Fabre, M. Rodríguez and F. Salles, “Dependability of COTS Microkernel-Based
Systems”, IEEE Transactions on Computers, vol. 51, no. 2, pp. 138-163, February 2002.
[14] J. Gray (Ed.) The Benchmark Handbook for Database and Transaction Processing Systems, San
Francisco, CA, USA: Morgan Kaufmann Publishers, 1993.
[15] D. Brock, “A Recommendation for High-Availability Options in TPC Benchmarks”,
http://www.tpc.org/information/other/articles/ha.asp
[16] R. K. Iyer and D. Tang, “Experimental Analysis of Computer System Dependability”, in FaultTolerant Computer System Design, (D. K. Pradhan, Ed.) pp. 282-392 (chapter 5), Upper Saddle River,
NJ, USA: Prentice Hall PTR, 1996.
[17] M. Kalyanakrishnam, Z. Kalbarczyk and R. K. Iyer, “Failure Data Analysis of LAN of Windows NT
Based Computers”, in Proc. 18th Int. Symposium on Reliable Distributed Systems (SRDS'99),
Lausanne, Switzerland, 1999, pp. 178-187 (IEEE Computer Society Press).
[18] J. Arlat, A. Costes, Y. Crouzet, J.-C. Laprie and D. Powell, “Fault Injection and Dependability
Evaluation of Fault-Tolerant Systems”, IEEE Transactions on Computers, vol. 42, no. 8, pp. 913-923,
August 1993.
[19] D. Costa, T. Rilho and H. Madeira, “Joint Evaluation of Performance and Robustness of a COTS
DBMS through Fault-Injection”, in Proc. Int. Conference on Dependable Systems and Networks
(DSN-2000), New York, NY, USA, 2000, pp. 251-260 (IEEE CS Press).
[20] H. Madeira, K. Kanoun, J. Arlat, Y. Crouzet, A. Johanson and R. Lindström, “Preliminary
Dependability Benchmark Framework”, DBench Project IST 2000-25425 Deliverable no. CF2,
Available at http://www.laas.fr/dbench/delivrables.html, 2001.
[21] K. Kanoun, M. Kaâniche and J.-C. Laprie, “Qualitative and Quantitative Reliability Assessment”,
IEEE Software, vol. 14, no. 2, pp. 77-86, mars 1997.
[22] K. Kanoun and M. Borrel, “Dependability of Fault-Tolerant Systems - Explicit Modeling of the
Interactions between Hardware and Software Components”, in IEEE International Computer
Performance & Dependability Symposium (IPDS'96), Urbana-Champaign, IL, USA, 1996, pp. 252261 (IEEE Computer Society Press).
[23] J. Arlat, K. Kanoun, H. Madeira, J. V. Busquets, T. Jarboui, A. Johansson and R. Lindström, “State of
the Art”, DBench Project IST 2000-25425 Deliverable no. CF1, Available at
http://www.laas.fr/dbench/delivrables.html, 2001.
[24] W.-L. Kao, R. K. Iyer and D. Tang, “FINE: A Fault Injection and Monitoring Environment for
Tracing the UNIX System Behavior under Faults”, IEEE Transactions on Software Engineering, vol.
19, no. 11, pp. 1105-1118, November 1993.
[25] K. Buchacker and V. Sieh, “Framework for Testing the Fault-Tolerance of Systems Including OS and
Network Aspects”, in Proc. High-Assurance System Engineering Symp. (HASE-2001), Boca Raton,
FL, USA, 2001, pp. 95-105.
[26] T. Jarboui, J. Arlat, Y. Crouzet and K. Kanoun, “Experimental Analysis of the Errors Induced into
Linux by Three Fault Injection Techniques”, in Proc. Int. Conf. on Dependable Systems and Networks
(DSN-2002), Washington, DC, USA, 2002 (IEEE CS Press).
[27] F. Raab, “Overview of the TPC Benchmark™ C: A Complex OLTP Benchmark”, in The Benchmark
Handbook for Database and Transaction Processing Systems, (J. Gray, Ed.) pp. 131-267, San
Francisco, CA, USA: Morgan Kaufmann Publishers, 1993.
[28] J. Durães, D. Costa and H. Madeira, “Accuracy of the Emulation of Software Faults by Machine-Code
Level Errors”, in Supplement of the Int. Conference on Dependable Systems and Networks (DSN2001), Göteborg, Sweden, 2001, pp. B.92-B.93 (Chalmers University of Technology, Göteborg,
Sweden).
[29] Saab Ericsson Space, “Microprocessor Thor, Product Information”, 1993.
[30] J. C. Baraza, J. Gracia, D. Gil and P. J. Gil, “A Prototype of a VHDL-based Fault Injection Tool”, in
Proc. Int. Symp. on Defect anf Fault Tolerance in VLSI Systems (DFT'2000), pp. 396-404.
Yamanashi, Japan, October 2000. (IEEE CS Press).
[31] K. Kanoun, J. Arlat, D. J. G. Costa, M. Dal Cin, P. Gil, J.-C. Laprie, H. Madeira and N. Suri,
“DBench – Dependability Benchmarking”, in Supplement of the 2001 Int. Conf. on Dependable
Systems and Networks (DSN-2001), Göteborg, Sweden, 2001, pp. D.12-D.15 (Chalmers University,
Göteborg, Sweden). See also http://www.laas.fr/dbench/delivrables.html
17