The DeLone and McLean Model of
Information Systems Success:
A Ten-Year Update
WILLIAM H. D E L O N E AND EPHRAIM R. McLEAN
WILLIAM DELONE is an Associate Professor of Information Systems and Chair of the
Information Technology Department at the Kogod School of Business at American
University in Washington, DC. Professor DeLone's primary areas of research include
the assessment of information systems effectiveness and value, the implementation
and use of information technology in small and medium-sized businesses, and the
global management of information technology. He has been published in various
journals including Information Systems Research, MIS Quarterly. DATABASE, Journal of Global Information Management, and Journal of Information Technology
Management. Professor DeLone earned a B.S. in mathematics from Villanova University, an M.S. in industrial administration from Cai'negie Mellon University, and a
Ph.D. in Computers and Information Systems from the University of California, Los
Angeles.
is a Regents' Professor and George E. Smith Eminent Scholar's
Chair in Information Systems in the Robinson College of Business at Georgia State
University, Atlanta. Prior to coming to Georgia State University in 1987, he was on
the faculty of the University of California, Los Angeles (UCLA) for 18 years. Dr.
McLean's research focuses on the management of information sen/ices, the value of
IS investments, and career issues for IS professionals. He has published over 125
papers in such journals as Information Systems Research, Journal of Management
Information Systems, MIS Quarterly, Management Science, Communications of the
ACM. DATABASE. Han>aril Business Review, Sloan Management Review, and others;
and his coauthored book. Information Technology for Management, now in its third
edition, is currently the second largest selling IS textbook in the world. Dr. McLean
earned his B.M.E. in mechanical engineering from Cornell University and his S.M.
and Ph.D. degrees from the Sloan School of Management at the Massachusetts Institute of Technology (MIT). He is also the Executive Director of the Association for
Information Systems (AISj and in 1999 was made a Eellow of the AIS.
EPHRAIM R. MCLE.'\N
Ten years ago, we presented the DeLone and McLean Information Systems (IS) Success Model as a framework and model for measuring the complexdependent variable in IS research. In this paper, we discuss many of the important IS
success research contributions of the last decade, focusing especially on research
efforts that apply, validate, challenge, and propose enhancements to our original model.
Based on our evaluation of those contributions, we propose minor i efinements to the
model and propose an updated DeLone and McLean IS Success Model. We discuss
the utility of the updated model for measuring e-commerce system success. Finally,
we make a series of recommendations regarding current and future measurement of
IS success.
ABSTRACT:
Journal of Mantif(emeiil
lutbrrntilioii Swieni.i/Spring
2fM).V Vol. 19. Ni>. 4, pp. 9—30.
Q:(XI.1 M.E. Sharpe. Int.
11742-1 222 / 2003 $9.50 + 0.00.
10
DELONE AND MCLEAN
evaluation of information systems, impact of information
technology, infonnation quality, information systems success, service quality, systems quality, use of information systems, user satisfaction.
KEY WORDS AND PHRASES:
(IS) success or effectiveness is critical to our understanding of the value and efficacy of IS management actions and IS
investments. In 1992, we published a paper [8] in which we attempted to bring some
awareness and structure to the "dependent variable"—IS success—in IS research. We
proposed a taxonomy and an interactive model (hereafter referred to as the "D&M IS
Success Model") as frameworks for conceptualizing and operationalizing IS success.
Since then, nearly 300 articles in refereed journals have referred to, and made use of,
this IS Success Model. The wide popularity of the model is strong evidence of the
need for a comprehensive framework in order to integrate IS researeh findings.
THE MEASUREMENT OF INFORMATION SYSTEMS
The D&M IS Success Model, though published in 1992, was based on theoretical
and empirical IS research conducted by a number of researchers in the 1970s and
1980s. The role of IS has changed and progressed during the last decade. Similarly,
academic inquiry into the measurement of IS effectiveness has progressed over the
same period. We reviewed more than 100 articles, including all the articles in Information Systems Research, Journal of Management Information Systems, and MIS
Quarterly since 1993 in order to inform this review of IS success measurement. The
purpose of this paper, therefore, is to update the D&M IS Success Model and evaluate
its usefulness in light of the dramatic changes in IS practice, especially the advent and
explosive growth of e-commerce.
The D&M IS Success Model
THE PRIMARY PURPOSE OF THE ORIGINAL DeLone
and McLean paper [8] was to synthesize previous re.search involving IS success into a more coherent body of knowledge and to provide guidance to future researchers. Based on the communications
research of Shannon and Weaver [431 ^"tl the information "influence" theory of Mason [31 ], as well as empirical management information systems (MIS) research studies from 1981-87. a comprehensive, mulfidimensional model of IS success was
postulated. Shannon and Weaver defined the technical level of communications as
the accuracy and efficiency of the communication system that produces information.
The semantic level is the success of the information in conveying the intended meaning. The effectiveness level is the effect of the information on the receiver. In the
D&M IS Success Model, "systems quality" measures technical success; "information
quality" measures semantic success; and "use, user satisfaction, individual impacts,"
and "organizational impacts" measure effectiveness success. In spite of the passage
of time since the Shannon and Weaver framework in 1949 and Mason's extensions in
1978, both appear as valid today as when we adopted them a decade ago.
THE DHLONE AND Md.KAN MODEL OF INFORMATION SYSTEM SUCCESS
11
Based on both process and causal considerations, these six dimensions of success
are proposed to be interrelated rather than independent. This has important implications for the measurement, analysis, and reporting of IS success in empirical studies.
A temporal, process model suggests that an IS is first created, containing various
features, which can be characterized as exhibiting various degrees of system and
information quality. Next, users and managers experience these features by using the
system and are either satisfied or dissatisfied with the system or its i riformation products. The use of the system and its information products then impacts or influences
the individual user in the conduct of his or her work, and these individual impacts
collectively result in organizational impacts. The resultant D&M IS Success Model is
reproduced in Figure I |8, p. 87].
In contrast to a process model, a causal or variance model studies the covariance of
the success dimensions to determine if there exists a causal relationship among them.
For example, higher system quality is expected to lead to higher user satisfaction and
use, leading to positive impacts on individual productivity, resulting in organizational
productivity improvements. The purpose of combining the success taxonomy with
the success model was to aid in the understanding of the possible causal interrelationships among the dimensions of success and to provide a more parsimonious exposition of the relationships. Unhappily, for some critics this combination has proved
troublesome, leading them to suggest a number of reformulations. These will be discussed later in this paper.
The primary conclusions of the original paper were:
1. The multidimensional and interdependent nature of IS success requires careful attention to the definition and measurement of each aspect of this dependent variable. It is important to measure the possible interactions among the
success dimensions in order to isolate the effect of various independent variables with one or more of these dependent success dimensions.
2. Selection of success dimensions and measures should be contingent on the
objectives and context of the empirical investigation; but, where possible, tested
and proven measures should be used.
3. Despite the multidimensional and contingent nature of IS success, an attempt
should be made to reduce significantly the number of different measures used
to measure IS success so that re.search results can be compiu^ed and findings
validated.
4. More field study research should investigate and incorporate organizational
impact measures.
5. Finally, "[tlhis success model clearly needs further development and validation before it could serve as a basis for the selection of appropriate IS measures" L8, p. 88].
Model Adoption
RESEARCH ADOPTION OF THE D & M IS SUCCESS MODEL has exceeded our expecta-
tions. A citation search in the summer of 2002 yielded 285 refereed papers in journals
12
DFI.ONE AND MrLEAN
System
Quality
Use
Individual
Impact
Information
Quality
User
Satisfaction
-1
Organizational
Impact
Figure J. D&M IS Success Model.
Reprinted by permission, W. DeLone and E. McLean, Information Systems Success: The
Quest for the Dependent Variable. Information Systems Research. 3(1), 1992, pp. 60-95.
Copyright 1992, The Institute of Management Sciences (now INFORMS), 901 Elkridge
Landing Road, Suite 400, Linthicum, MD 21090 USA.
and proceedings that have referenced the D&M Model during the period 1993 to
mid-2002. Many of the.se articles positioned the measurement or the development of
theirdependentvariable(s) within the context ofthe D&M IS Success framework. By
using the model as a common framework for reporting and comparing research work
involving IS success or effectiveness, we believe one of the primary purposes of the
original article has been achieved.
Although many of the cited articles tended to justify their empirical measurement
of IS success by citing the D&M IS Success Model, some of them failed to heed our
cautions. Some researchers have used the model to support their chosen success variable rather than to inform the development of a more comprehensive success construct. They overlooked the main conclusion of the article—that IS success is a
multidimensional and interdependent construct—and that it is therefore necessary to
study the interrelationships among, or to control for, those dimensions. "Researchers
should systematically combine individual measures from the IS success categories to
create a comprehensive measurement instrument" [8, pp. 87-88]. Although these
authors did not choose to measure (or control for) the various dimensions of IS success, a number of other researchers have used multidimensional measures of IS success in their empirical studies and have analyzed the interrelationships among them.
Some of these studies are summarized in the next section.
Model Validation
UNLtKE A PROCESS MODEL, which merely states that B follows A, a causal model
postulates that A (.au.^ej B; that is, increasing A will cause B to increase (or decrease).
THE DELONE AND MCLEAN MODEL OF INHORMATION SYSTEM SUCCESS
13
In the 1992 article we proposed such interrelationships among the dimensions in our
model; but we did not test them empirically. Since 1992, a number of studies have
undertaken empirical investigations of the multidimensional relationships among the
measures of IS success.
Empirical testing and validation of the D&M IS Success Model was the primary
purpose of two research studies [38,41 ]. Seddon and Kiew [41] surveyed 104 users
of a recently implemented, university accounting system and found significant relationships between "system quality" with "user satisfaction" and "individual impact,"
between "infonnation quality" with "user satisfaction" and "individual impact," and
between "user satisfaction" and "individual impact." Rai et al. [381 performed a
goodness-of-fit test on the entire D&M IS Success Model based on survey responses
from 274 users of a university student IS. The study found that some goodness-of-fit
indicators were significant but others were not. However, all of the path coefficients
among success dimensions of the D&M IS Success Model were found to be significant.
Other empirical studies explicitly tested the associations among tbe measures identified in the D&M IS Success Model [10, 12, 14, 16, 22. 50]. Yet other empirical
studies have implicitly tested the model by investigating multiple success dimensions
and their interrelationships [II, 17, 48, 52, 54, 57, 58, 60|. Figure 2 displays the
D&M IS Success Model and the relationships confirmed (or not confirmed) in the 16
empirical studies cited above. These .studies were selected based on the fact that they
used multidimensional success constructs and they measured the association among
the success constructs. In the following paragraphs, these empirical results are summarized by the success links that were tested. Links with the strongest empirical
support are discussed first.
System Use—^Individual Impacts
Seven of the 16 studies in Eigure 2 [12, 14, 16, 48, 52, 54. 60] tested the association
between '"system use" and "individual impacts" and the association was found to be
significant in each of the studies. System use was typically voluntary and was measured
as frequency of use, time of use, number of accesses, usage pattern, and dependency.
Individual impacts were measured in terms of job performance and decision-making
performance.
System Quality—Individual Impacts
All five studies [10, 12, 41, 50. 57] that tested the direct association between "system
quality" and "individual impacts" found those associations to be statistically significant. System quality was measured in terms of ease-of-use, functionality, reliability,
flexibility, data quality, portability, integration, and importance. Individual impacts
were measured as quality of work environment and job perfonnance.
14
III
6
o ^
-a
«
•a
gbari£
thas a
V
1601:
and Thor
» oo
t
^^15
.S ta « .M
Q c U "
S ^5 00 ^
.PC ^
S OJ
THE DELONB AND MCLEAN MODEL OF INFORMATION SYSTEM SUCCESS
15
Information Quality—Individual Impacts
The four studies [10, 41, 50, 57] that tested the relationship between "information
quality" and "individual impacts" found the association to be significant. Information
quality was measured in terms of accuracy, timeliness, completeness, relevance, and
consistency. Individual impact was measured in terms decision-making performance,
job effectiveness, and quality of work.
Other Links
With one exception, all the other links or associations in the D&M IS Success Model
were empirically validated. The one empirical study that found the associations not
significant was a survey of Dutch managers [II], where the association between system use and organizational revenues and profitability was not statistically significant.
In conclusion, 36 of the 38 success factor associations that were empirically tested
in the 16 studies summarized in Eigure 2 were found to be significant. Taken as a
whole, these empirical studies give strong support for the proposed associations among
the IS success dimensions and help to confirm the causal structture in the model.
Model Issues
that have tested and validated the D&M IS Success Model, several articles have been published that challenge, critique, or extend
tbe model itself. On balance, these articles have contributed to a better understanding
of success and its dimensions. These articles and the issues tbey raise to the D&M IS
Success Model are summarized below.
IN ADDITION TO THE MANY PAPERS
Process Versus Causal Models
The D&M IS success taxonomy and its six success categories are based on a process
model of IS |43|. fnaddition, we argue that the six dimensions are interrelated, resulting in a success model that indicates that causality flows in the same direction as the
information process. However, citing an earlier paper by Newman and Robey i35|,
Seddon argues that "the boxes and arrows in variance- and process-model diagrams
represent quite different concepts and cannot be combined meaningfully in one
model. . . . Unfortunately, combining variance and process models is exactly what
[DeLone and McLean have] attempted to do" [40]. Seddon further argues that DeLone
and McLean have "attempted to combine both process and causal explanations of IS
success in their model. After working with this model for some years, it has become
apparent that the inclusion of both variance and process interpretations in their model
leads to so many potentially confusing meanings" [40, p. 240J. Seddon goes on to
propose a respecified variance model of IS success.
16
DELONE A N D MCLEAN
We agree with Seddon's premise that the combination of process and variance
interpretations of IS success in one model can be confusing. However, we believe
that Seddon's reformulation of the D&M Model into two partial variance models
[40, p. 245] unduly complicates the success model, defeating the intent of the original model.
The creation of the D&M IS Success Model was driven by a process understanding
of IS and their impacts. This process model has just three components: the creation of
a system, the use of the system, and the consequences of this system use. Each of
these steps is a necessary, hut not sufficient, condition for the resultant outcome(s).
For instance, without system use, there can be no consequences or benefits. However,
with system use, even extensive use. which is inappropriate or ill-informed, there
may also be no benefits. Thus, to understand fully the dimensions of IS success, a
variance model is also needed. Thus, as Seddon himself pointed out [40, 42], the
application of our model to empirical research also requires a contextual variance
specification of the model. Here too there are three components: the first is production, the second is use, and the third is net benefits. The only argument is whether
these two necessary dimensions can be combined into one model. Along with Seddon,
we believe that they can; only our formulations are different.
System Use as a Success Measure
Seddon |40) further argues for the removal of "system use" as a success variable in
the causal success model, claiming that use is a behavior, appropriate for inclusion in
a process model but not in a eausal model. He argues that use must precede impacts
and benefits, but it does not cause them. We disagree. We believe that system usage is
an appropriate measure of success in most cases.
The problem to date has been a too simplistic definition of this complex variable.
Simply saying that more use will yield more benefits, without considering the nature
of this use, is clearly insufficient. Researchers must also consider the nature, extent,
quality, and appropriateness of the system use. The nature of system use could be
addressed by determining whether the full functionality of a system is being used for
the intended purposes. Young and Benamati [59]. for example, suggest that full functional use of an e-commerce system should include informational use, transactional
use, and customer service use. With regard to the extent of use, Lassila and Brancheau
[27] identify various states of systems utilization based on the use or nonuse of basic
and advanced system capabilities. Simply measuring the amount of time a system is
used does not properly capture the relationship between usage and the realization of
expected results. On the other hand, it can be argued that declining usage may be an
important indication that the anticipated benefits are not being realized.
The rejection of system use as a success variable when system usage is mandatory
is also flawed for the reasons cited above. Even when use is required, variability in
the quality and intensity of this use is likely to have a significant impact on the realization of the system benefits. Furthermore, no system use is totally mandatory. At
some level of the organization, an executive or management committee has chosen to
THE DELONE AND MCLEAN MODEL OF INFORMATION SYSTEM SUCCESS
17
implement a system and require employees to use it. Thus, whereas usage of a system
may be mandatory at one level, the continued adoption and use of the system itself
may be wholly voluntary, based upon management judgment, at a higher level. Management always has the option of discontinuing a system that is not providing the
desired results and benefits.
System usage continues to be used as u dependent variable in a number of empirical
studies and continues to be developed and tested by IS researchers 111, 12. 14, 16, 17,
38, 47, 48, 52, 54, 60]. System use has taken on new importance in e-commerce
success measurements where customer use is voluntary and essential to desired outcomes [7, 29, 36|. "While most studies that follow D&M replace ihe Use box with
Usefulness . . . , we prefer to maintain U.se as in the original work. In e-commerce
systems Use is largely voluntary" [33, p. 6). We agree with these IS researchers and
believe that use. especially informed and effective use. will continue to be an important indication of IS success for many systems.
Role of Context
Several researchers have commented on the difficulty of applying the D&.M IS Success Model in order to define and operationalize IS success in specific research contexts. This was not unexpected: "This success model clearly needs further development
and validation before it could serve as a basis for the selection of appropriate IS
measures" [8, p. 88]. Jiang and Klein [20] found that users prefer different success
measures, depending on the type of system being evaluated. Whyte et al. found that
"there are important differences deriving from organizational, user, and systems variations which can modify the view as to which attributes (success measures) are important" [55, p. 65]. Seddon et ai. [42] make an important contribution by proposing a
two-dimensional matrix for classifying IS effectiveness measures based on the type
of system studied and on the stakeholder in whose interest the IS is being evaluated.
In this regard, we completely agree. As stated in the 1992 article, "no single variable
is Intrinsically better than another, so the choice of success variables is often a function oi'the objective of the study, the organizational context... etc." [8, p. 80, emphasis added [.
Independent Versus Dependent Variables
Many of the suggested improvements to the D&M IS Success Model flow from a
confusion between what is an independent variable and what is part of the dependent
variable, IS success. "User involvement" and "top management support" are but two
examples of suggested additions to the D&M Model; yet these are clearly variables
that may cause success rather than being a part o/success. "Investing in ERP" may
(or may not) lead to improved "information quality" (an aspect of IS success), but the
former is an independent variable whereas the latter is part of the dependent variable.
It is essential that IS researchers distinguish between the management control variables and the desired results in terms of quality, use satisfaction, and impacts.
18
D E L O N E AND McLEAN
Model Extensions
Service Quality
in the mid-1980s placed IS organizations
in the dual role of information provider (producing an information product) and service provider (providing support for end user developers). Pitt et al. observed that
"commonly used measures of IS effectiveness focus on the products rather than the
services of the IS function. Thus, there is a danger that IS researchers will niismeasure
IS effectiveness if they do not include in their assessment package a measure of IS
service quality" [37, p. 173]. Other researchers have agreed with this, citing the need
for a service quality measure to be a part of IS success [25, 28, 56].
Researchers who have argued that service quality be added to the success model
have applied and tested the 22-item SERVQUAL measurement instrument from marketing [25, 37] to an IS context. This instrument uses the dimensions of tangibles,
reliability, responsiveness, assurance, and empathy to measure service quality. Some
sample SERVQUAL instrument items include:
THE EMERGENCE OF END USER COMPUTING
•
•
'
•
•
"IS has up-to-date hardware and software" (tangible);
"IS is dependable" (reliability);
"IS employees give prompt service to users" (responsiveness);
"IS employees have the knowledge to do their job well" (assurance); and
"IS has users' best interests at heaif (empathy).
Van Dyke et al. [53 [ challenged this SERVQUAL metric, identifying "problems
with the reliability, discriminant validity, convergent validity, and predictive validity
of the measure. . . . [Ejurther work is needed in the development of measures for
assessing the quality of information services." Recently, Jiang et al.'s [21] empirical
study among 168 users and 168 IS professionals concluded that the SERVQUAL
measure is a valuable analytical tool for IS managers. The study found high convergent validity for the reliability, responsiveness, assurance, and empathy of the
SERVQUAL scales and found acceptable levels of reliability and discriminant validity among the reliability, responsiveness, and empathy scales.
Whereas we agree that the SERVQUAL metric needs continued development and
validation, we nevertheless believe that "service quality," properly measured, deserves
to be added to "system quality" and "information quality" as components of IS success. Although a claim could be made that "service quality" is merely a subset of the
model's "system quality," the changes in the role of IS over the last decade argue for
a separate variable—the "service quality" dimension.
Of course, each of these quality dimensions will have different weights depending
upon the level of analysis. To measure the success of a single system, "infonnation
quality" or "system quality" may be the most important quality component. For measuring the overall success of the IS department, as opposed to individual systems,
"service quality" may become the most important variable. Once again, context should
dictate the appropriate specification and application of the D&M IS Success Model.
THE D E L O N E AND Mi LEAN MODEL OF INFORMATION SYSTEM SUCCESS
19
Net Benefits
As the "impacts" of IS have evolved beyond the immediate user, researchers have
suggested additional IS impact measures, such as work group impacts [ 18, 34], interorganizational and industry impacts [5, 6], consumer impacts [3, 15], and societal
impacts [401. Clearly, there is a continuum of ever-increasing entities, from individuals to national economic accounts, which could be affected by IS activity. The choice
of where the impacts should be measured will depend on the system or systems being
evaluated and their purposes. Rather than complicate the model with more success
measures, we prefer to move in the opposite direction and group all the "impact"
measures into a single impact or benefit category called "net benefits." Although, for
some studies, such finer granularity may be appropriate, we resisted such further
refinements for the sake of parsimony. This is discussed further in the Analysis and
Recommendations section.
Measurement Enhancements
1 THROUGH 6 IN THE 1992 ARTICLE [8. pp. 65-83] listed the numerous success measures within each success category that had been used in previous empirical
studies. We called for "a significant reduction in the number of dependent variable
measures so that research results can be compared" [8, p. 80]. Since then, several
studies have developed and tested survey instruments, which measure one or more of
these six success constructs.
TABLES
Based on a comprehensive literature review, Mirani and Lederer [32] developed a
33-item instrument to measure organizational benefits derived from IS projects. Their
measurement framework consisted of three categories of organizational benefits: strategic, infonnational, and transactional. The proposed instrument was empirically tested
in a survey of 200 IS managers and systems analysts. The results showed strong
evidence of discriminant validity. Further analysis identified three subdimensions for
each of the benefit categories. Strategic benefits were further subdivided into competitive advantage, alignment, and customer-relations benefits. Informational benefits included information access, information quality, and infoniiation flexibility
subdimensions; and finally, transactional benefits included communication efficiency,
systems development efficiency, and business efficiency subdimensions. We believe
that validated instruments, such as this, that measure IS benefits are an important
contribution to IS success measurement.
In a conceptual paper, Martinsons et al. [30] suggest an adaptation of the Kaplan and
Norton "Balanced Scorecard" (BSC) [23] approach for the measurement of organizational performance. The BSC consists of four performance perspectives: the financial
perspective, the customer perspective, the internal business process perspective, and
the learning and growth perspective. Applied to an IS context, the authors propose a
balanced IS scorecard to include a business-value measurement dimension, a userorientation dimension, an internal-process dimension, and a future readiness dimension. The authors then suggest specific measures related to each IS BSC dimension.
20
DtLONE AND McLEAN
For example, cost control, revenue generation, strategic alignment, and return on investment are among the measures suggested for the business-value dimension.
Based on a literature review, Torkzadeh and Doll |52] developed a four-factor, 12itein instrument for measuring the individual impact of IS. A survey of 409 end users
from 18 different organizations was used to test the measurement instrument. The
overall reliability of the 12-item scale was 0.92. The empirical evidence also supports
the convergent and discriminant validity of the instrument. The resulting individual
impact dimensions are:
• Task productivity—the extent to which an application improves the user's output per unit of time;
• Task innovation—the extent to which an application helps users create and try
out new ideas in their work:
• Customer satisfaction—the extent to which an application helps the user create
value for the firm's internal or external customers; and
• Management control—the extent to which the application helps to regulate work
processes and performance.
Using a 24-item impact measurement instrument, Jiang and Klein [20] surveyed
113 managers regarding systems impacts across three different .system types: transaction processing systems (TPS), information reporting systems (IRS), and decision
support systems (DSS). Their study found that users value decision quality impacts in
DSS, but otherwise, value systems performance impacts for TPS and IRS. These findings suggest that different impact measures are appropriate for different types of systems.
There continues to be much use—and discussion—of the User Information Satisfaction (UIS) instrument 12, 19J. Saarinen [39J has developed an expanded user satisfaction instrument that adds development process and IS impact dimensions to the
use and product quality dimensions of the traditional LJIS instrument |2, 19|. His
four-factor, 52-item user satisfaction instrument was developed and tested based on
48 completed IS development projects. Reliability and validity tests found the four
satisfaction constructs had acceptable reliability and satisfactory validity. The resulting instrument may serve as a more comprehensive perceptual surrogate for IS success. Li [28] also proposed an extended information satisfaction instrument but did
not test the instrument.
Other authors have identified problems with UIS |51 ] and have proposed an alternate satisfaction measure [44]. In spite of these criticisms, UIS continues to be the
most commonly used and developed success measure; but, when used alone, it cannot fully measure IS success.
As indicated earlier, systems use continues to be a popular success measure [17, 26,
47, 48]. However, Straub et al. [46] studied 458 users of a voice mail system and
found that self-reported systems usage and computer-recorded usage were not correlated. Their findings suggest that self-reported system usage and computer-recorded
usage should both be measured in empirical studies becau.se the two do nol necessarily correlate u'ith one another.
THE D E L O N E AND McLEAN MODEL OF INFORMATION SYSTEM SUCCESS
21
Based on a literature review, Doll and Torkzadeh [9] developed a multidimensional
measure of systems usage based on the nature and purpose of a system. A 30-item
system usage instrument was tested using 409 computer users from 18 organizations.
The 30 items measure three underlying systems usage constructs: use for decision
support, use for work integration, and use for customer service. The empirical results
provided evidence of the instrument's reliability, validity, and general applicability.
Researchers should consider adopting and applying this more comprehensive systems usage instrumenL
Agai-wal and Prasad [ 1 j studied both initial system usage and intentions of future
use and found that different factors affected initial use versus future use of the World
Wide Web. Similarly. Karahanna et al. [24] found different factors were associated
with intention to use windows between potential adopters and continuing users. These
two empirical studies demonstrate that early use and continued use can differ.
System use is clearly a key variable in understanding IS success; but, too frequently,
simple usage variables are used to measure this complex construct. More research
such as cited above is needed to refine the multidimensionality of systems usage.
Information quality has proven to be strongly associated with system use and net
benefits in recent empirical studies [38. 54. 57] and especially in the context of
e-commerce systems [7, 29, 33, 36,49]. According to Molla and Licker. '•[A]lthough
information has long been considered as an important asset to modern business,
e-commerce has elevated content, i.e. information . . . to higher levels of significance" [33, p. 7]. Information quality measures that have been used in recent e commerce studies [7, 33,36] include accuracy, relevance, understandability, completeness,
currency, dynamism, personalization, and variety. Researchers are strongly encouraged to include information quality measures as a critical dimension of their success
measurement construct.
Other Success Frameworks
to critique or modify the D&M IS
Success Model. Some have developed and proposed alternate frameworks for measuring IS effectiveness. Grover et al. used an alternative, theoretically based perspective (theory of organizational effectiveness) "to build a theoretically-based construct
space for IS effectiveness which complements and extends the [DeLone & McLean]
IS Success Model" [13, p. 178]. Based on unit-of-analysis and evaluation-type context dimensions, the authors created six IS effectiveness categories. The six effectiveness classes are infusion measures (i.e., "organizational impacts" in the D&M IS
Success Model), market measures (not covered in the D&M IS Success Model), economic measures (i.e., "organizational impacts"), usage measures (i.e., "system use"),
perceptual measures (i.e., "user satisfaction"), and productivity measures (i.e., "individual impact"). Their framework considers "system quaiity" and "information quality" to be antecedent effectiveness constructs, whereas the D&M IS Success Model
considers them to be important dimensions of success itself. In summary, the Grover
et al. [13] IS effectiveness framework serves to validate the D&M IS Success Model
NOT ALL OF THE RESEARCHERS HAVE ATTEMPTED
22
D E L O N E AND McLEAN
from a theoretical perspective and suggests an area for extension, namely, market
impacts. We include market or industry impacts in our updated model described later
in this paper.
Smithson and Hirschheim [45] proposed a cnneeptual framework for IS evaluation
and demonstrated its usefulness in practice by applying the framework to the evaluation of an outsourcing situation. Their framework presents various theoretical bases
for IS evaluation organized into three "zones" of evaluation: efficiency, effectiveness,
and understanding. Appropriate constructs or metrics could be drawn from the literature stream associated with each conceptual base; for example, software metrics, organizational behavior, sociology, cognitive psychology, and so on. This fi-amework
includes evaluation areas that overlap the D&M success dimensions, including hardware and software metrics ("system quality"), system usage, user satisfaction, costbenefit analysis, and so on, but also suggests many other theoretical sources of IS
evaluation measures. The authors provide a framework that is a source for identifying
and developing IS evaluation measures rather than a single framework of success
dimensions and their interrelationships (i.e., the D&M TS Success Model). Their framework does not specify actual success constructs and related measures. This makes the
framework difficult to apply in practice. However, it does offer the researcher an
alternative theoretical framework for developing IS evaluation schemes.
Analysis and Recommendations
A s DISCUSSED EARLIER, IT NOW SEEMS APPROPRIATE to add a third dimension, "service
quality," to the two original system characteristics, "systems quality" and "information
quality." Conversely, as discussed earlier, it appears more parsimonious to combine
"individual" and "organizational impacts" into a single variable, "net benefits."
This new variable, "net benefits," immediately raises three issues that must be taken
into account: what qualities as a "benefit"? for whom? and at what level of analysis?
In the original formulation of the D&M Model, the term "impact" was used. Seddon
[40] used "consequences" and "net benefits" in his characterization of the outcomes.
We have come to prefer the term "net benefits" ourselves because the original term
"impacts" may be positive or negative, thus leading to a possible confusion as to
whether the results are good or bad. Also, the inclusion of "net" in "net benefits" is
important because no outcome is wholly positive, without any negative consequences.
Thus, "net benefits" is probably the most accurate descriptor of the final success
variable.
The second issue of concern is: benefits for whom—the designer, the spon.sor, the
user, or others? Different actors, players, or stakeholders may have different opinions
as to what constitutes a benefit to them [42]. Thus, it is impossible to define these "net
benefits" without first defining the context or frame of reference. The fact that the
D&M Model does not define this context is a matter of detail, not of oversight. The
focus of any proposed study must be defined. Our model may be useful to both
Microsoft and the user community, but each may have a very different definition of
what constitutes net benefits and thus IS success.
THE DELONE AND McLEAN MODEL OF INFORMATION SYSTEM SUCCESS
23
Einally, the level of analysis must be addressed [4, 42]. Are the benefits to be measured from the individual's perspective, his or her employer, or that of the industry or
of the nation? Collapsing "individual" and "organizational impacts" into a single
variable, "net benefits," does not make the problem go away. It merely transfers the
need to specify the focus of analysis to the researcher.
Based on these considerations, we have updated the original D&M IS Success Model
as a foundation for framing future IS empirical research.
The Updated D&M IS Success Model
since our original paper, and based on changes
in the role and management of information systems, we have updated our original
success model. The updated model is presented on Figure 3.
As discussed earlier, quality has three major dimensions; "information quality,"
"systems quality." and "service quality." Each should be measured—or controlled
for—separately, because singularly or jointly, they will aftect subsequent "use" and
"user satisfaction."
Given the difficulties in interpreting the multidimensional aspects of "use"—mandatory versus voluntary, informed versus uninformed, effective versus ineffective,
and so on—we suggest "intention to use" may be a worthwhile alternative measure in
some contexts. "Intention to use" is an attitude, whereas "use" is a behavior. Substituting the lonner for the latter may resolve some of the process versus causal concerns that Seddon (1997) has raised. However, attitudes, and their li nks with behavior,
are notoriously difficult to measure; and many researchers may cboose to stay with
"use," but hopefully with a more informed understanding of it.
BASED ON RESEARCH CONTRIBUTIONS
As was true in the original formulation of the D&M Model, "use" and "user satisfaction" are closely interrelated. "Use" must precede "user satisfaction" in a process
sense, but positive experience with "use" will lead to greater "user satisfaction" in a
causal sense. Similarly, increased "user satisfaction" will lead to increased "intention
to use," and thus "use."
As a result of this "use" and "user satisfaction," certain "net benefits" will occur. If
the IS or service is to be continued, it is assumed tbat the "net benefits" from the
perspective of the owner or sponsor of the system are positive, thus inlluencing and
reinforcing subsequent "use" and "user satisfaction." These feedback loops are still
valid, however, even if the "net benefits" are negative. The lack of positive benefits is
likely to lead to decreased use and possible discontinuance of the system or of the IS
department itself (e.g., wholesale outsourcing). The challenge for the researcher is to
define clearly and carefully the stakeholders and context in which "net benefits" are
to be measured.
The updated D&M IS Success Model includes arrows to demonstrate proposed
associations atnong success dimensions in a process sense, but does not show positive or negative signs for those associations in a causal sense. The nature of these
causal associations should be hypothesized within the context of a particular study.
Eor example, in one instance a high-quality system will be associated with more use,
24
D Q L O N E A N D MfLEAN
INFORMATION
QUALrrv
INTENTION
TO USE
USE
I
SYSTEM QUALITY
NET
BEN E n T S
USER
SATISFACTION
SERVICE
QUALITY
J
Figure 3. Updated D&M IS Success Model
more user satisfaction, and positive net benefits. The proposed associations would
then all be positive. In another circumstance, more use of a poor quality system would
be associated with more dissatisfaction and negative net benefits. The proposed associations would then be negative.
E-Commerce Success
and the Internet in particular, is having a
dramatic impact on business operations. Companies are making large investments in
e-commerce applications but are hard-pressed to evaluate the success of their e-commerce systems. IS researchers have turned their attention to developing, testing, and
applying e-commerce success measures [7, 29, 33, 36. 49]. Molla and Licker [33|
proposed an e-commeree success model based on the D&M IS Success Model. This
section demonstrates how the updated D&M IS Success Model can be adapted to the
measurement challenges of the new e-commerce world.
INFORMATION TECHNOLOGY IN GENERAL,
As a powerful communications and commerce medium, the Internet is a communication and IS phenomenon that lends itself to a measurement framework (i.e.. the
D&M IS Success Model) that is built on communication theory (e.g.. Shannon and
Weaver [43J). Within the e-commerce context, the primary system users are customers or suppliers rather than internal u.sers. Customers and suppliers use the system to
make buying or selling decisions and execute business transactions. These electronic
decisions and transactions will then impact individual users, organizations, industries, and even national economies. This communications and comnierce process fits
nicely into the updated D&M IS Success Model and its six success dimensions.
• "System quality," in the Internet environment, measures the desired characteristics of an e-commerce system. Usability, availability, reliability, adaptability.
THE DELONE AND McLEAN MODEL OF INFORMATION SYSTEM SUCCESS
•
•
•
•
•
25
and response time (e.g., download time) are examples of qualities that are valued by users of an e-commerce system.
"Information quality" captures the e-commerce content issue. Web content should
be personalized, complete, relevant, easy to understand, and secure if we expect
prospective buyers or suppliers to initiate transactions via the Internet and return
to our site on a regular basis.
"Service quality," the overall support delivered by the service provider, applies
regardless of whether this support is delivered by the IS department, a new organizational unit, or outsourced to an Internet service provider (ISP). Its importance is most likely greater than previously since the users are now our customers and poor user support will translate into lost customers and lost sales.
"Usage" measures everything from a visit to a Web site, to navigation within the
site, to information retrieval, to execution of a transaction.
"User satisfaction" remains an important means of measuring our customers'
opinions of our e-commerce system and should cover the entire customer experience cycle from information retrieval through purchase, payment, receipt, and
service.
"Net benefits" are the most important success measures as they capture the balance of positive and negative impacts of the e-commerce on our customers,
suppliers, employees, organizations, markets, industries, economies, and even
our societies. Have Internet purchases saved individual consumers time and
money? Have the benefits such as laiger markets, supply chain efficiencies, and
customer responsiveness yielded positive net benefits for an organization? Have
countries' investments in e-commerce infrastructure yielded a net positive growth
in the gross national product? Have societal investments in e-commerce infrastructure and education reduced poverty? "Net benefits" measures must be determined by context and objectives for each e-commerce investment. Thus, there
will be a variety of e-commerce "net benefits" measures, but many will be the
same ones that have been developed and tested for IS investments in general.
"Net benefits" success measures are most important, but they cannot be analyzed
and understood without "system quality" and "information quality" measurements.
For example, within the e-commerce environment, the impact of a Web site design on
customer purchases cannot be fully understood without an evaluation of the usability
of the Web site and the relevance for purchasing decisions of the information that is
provided to the prospective purchaser.
Table 1 demonstrates how the six dimensions of the updated D&M IS Success
Model can be used as a parsimonious framework to organize the various success
metrics identified in the IS and e-commeree literature.
Summary and Conclusions
TEN YEARS AGO WE PUBLISHED AN IS SUCCESS FRAMEWORK
in order to integrate the
research work on IS success that had been done up to that point and to provide
26
D E L O N E AND McLEAN
Table I. E-Commerce Success Metrics
Systems quality
• Adaptability
• Availabiiity
• Reliability
• Response time
• Usability
Information quality
• Completeness
• Ease of understanding
• Personalization
• Relevance
• Security
Service quality
• Assurance
• Empathy
• Responsiveness
Use
• Nature of use
• Navigation patterns
• Number of site visits
• Number of transactions executed
User satisfaction
• Repeat purchases
• Repeat visits
• User surveys
Net benefits
• Cost savings
• Expanded markets
• Incremental additional sales
• Reduced search costs
• Time savings
directions for future research. We presented an IS success model that attempted to
capture the multidimensional and interdependent nature of IS success. The hundreds
of research works that have applied, developed, challenged, or validated the original
model speak to the need for a common approach to success measurement and to the
value of a model for framing research designs. The succeeding ten years have seen
tremendous progress in terms of the impacts of IS on business and society as well as
progress in IS re.search. In light of this progress and change, we felt compelled to
review, evaluate, and update our success model.
Considering the recent research studies that both validate and support our model as
well as those that challenge it, we conclude that our original model and related con-
THE DELONE AND Mt LEAN MODEL OF INFORMATION SYSTEM SUCCESS
27
elusions [8] still form a sound basis for IS success measurement even in the e-commerce environment. We believe that our proposed changes in the updated D&M IS
Success Model are largely changes in degree, not in kind. The addition of "service
quality" and the collapsing of "individual impacts" and "organizational impact" into
"net benefits" still preserve the parsimonious nature of the model.
Based on our review of the research experience of the last decade, we draw the
following conclusions:
1. Many empirical studies have validated the original model and its interrelationships, whereas other studies have recommended enhancements to the original
model. Based on these contributions, we propose an updated D&M IS Success
Model to serve as a foundation for the positioning and comparing of IS empirical research. The model should continue to be tested and challenged. The
changes introduced in this paper are examples of this continued growth and
refinement. We encourage others to join in this effort.
2. The updated D&M IS Success Model is a useful model for developing comprehensive e-commerce success measures as demonstrated in Table I.
3. We recommend that "service quality" be added as an important dimension of
IS success given the importance of IS support, especially in the e-commerce
environment where customer service is crucial.
4. The complex, multidimensional, and interdependent nature of IS success requires careful attention to the definition and measurement of each dimension
of this dependent variable. It is important to measure the possible interactions
among these success dimensions in order to isolate the effect of various independent variables witb one or more of these dependent success dimensions.
The updated D&M IS Success Model in Figure 3 presents the interdependent
relationships that should continue to be considered and tested,
5. Eor each research endeavor, the selection of IS success dimensions and measures should be contingent on the objectives and context of the empirical investigation, but, where possible, tested and proven measures should be used.
The Seddon et al. [42| context matrix is a valuable reference for selection of
success measures based on context.
6. Despite the multidimensional and contingent nature of IS success, an attempt
should be made to reduce significantly the number of measures used to measure IS success so that research results can be compared and findings validated. Some good progress has been made in this area as noted in the
Measurement Enhancements section of this paper. Where possible, we advocate the application of existing, validated measures rather than the development of new measures.
7. With the growth of management support systems and the advent and development of e-commerce systems, voluntary systems use is more common today than it was a decade ago. We, therefore continue to advocate the inclusion
of "System Use" as a critical dimension of IS success measurement. Actual
use measures should be preferred to self-reported use measures. Also, usage
28
D E L O N E AND McLEAN
measures should capture the richness of use as a system phenomenon including the nature, level, and appropriateness of use, and should not simply measure the frequency of use.
8. Finally, more field-study research should investigate and incorporate "Net
Benefits" measures. Yuthas and Young support this conclusion: •'[E]xamining
satisfaction and usage measures is not an acceptable alternative to measuring
performance [i.e.. Net Benefits] directly. Although the three variables are correlated, the relationships between them are not sufficiently strong to warrant
their use as substitutes for one another" [60, p. 121]. Good progress has been
made in the development and testing of "Net Benefits" measures on the individual, group, firm, industry, and national levels.
REFERENCES
1. Agarwal, R., and Prasad, J. The role of innovation characteristics and perceived
voluntariness in the acceptance of information technologies. Decision Sciences 28 3 (]991)
557-580.
2. Bailey, J.E., and Pearson, S.W. Development of a tool for measuring and analyzing
computer user satisfaction. Management Science, 29. 5 (\983), 530-545.
3. Brynjolfsson, E. The contrihution of information technology to consumer welfare Informathm Sy.steim Re.search. 7, 3 (1996), 281-300.
4. Chan, Y.E. IT value: The great divide between qualitative and quantitative and individual
and organizational mea.sures. Journal ()fMana}>ement Information Systems. 16 4 (Sprina 2000)
225-261.
"
Vp
6
7,
5. Clemons, E.K., and Row. M.C. Limits lo interfirm coordinalion through information
technology: Results of a field study in consumer goods packaging distribulion. Journal of
Management Information Sy.stems, 10. 1 (Summer 1993), 73-95.
6. Clemons. E.K.: Reddi. S.P: and Row, M.C. The impacl of information technology on the
organization of economic activity: The "move to the middle" hypothesis. Journal ofManatiemcnl Information Systems, 10. 2 {Fall 1993), 9-35.
7. D'Ambra, J., and Rice R.E. Emerging factors in user evaluation of the Worldwide Web.
Information & Management, 38. 6 (2001), 373-384.
8. DeLone, W.H.. and McLean, E.R. Information systems success: The quest for the dependent variable. Informalion 5v.v/cm.v Research. 3. 1 (1992). 60-95.
9. Doll, W.J., and Torkzadeh, G. Developing a multidimensional measure of systems use in
an organizationa] context. Information <SL Management. 33. 4 (1998), 171-185.
10. Elezadi-Amoli, J.. and Earhoomand, A.E. A sLructural model of end user computing
satisfaction and user performance. Information & Management. 30. 2 (1996). 65-73.
11. Gelderman, M. The relation between user satisfaction, usage of information .systems,
and performance, tnfonmilion & Management, 34. I (1998), 11-I8.
12. Goodhue, D.L., and Thompson, R.L. Task-technology fit and individual performance
MIS Quarterly. 19. 2 (1995), 213-233.
13. Grover, G.; Jeong, S.R.; and Segars, A.H. Information systems effectiveness: The construct space and patterns of application. Infonmition & Management. 31. 4 (1996), 177-191.
14. Guimaraes, T, and Igbaria, M. Client/server system success: Exploring the human side
Decision Sciences. 2S. 4 (1997). 851-875.
15. Hitt, L., and Brynjolfsson, E. The three faces of IT value: Theory and evidence. In J.I.
DeGross, S.L. Huff, and M.C. Munro (eds.). Proceedings of the International Conference on
hiformation Systems. Atlanta, GA: Association for Information Systems, 1994, pp. 263-278.
16. Igbaria, M., and Tan, M. The consequences of the information technology acceptance on
subsequent individual performance. Information & Management. 32. 3 (1997), 113-121.
17. Igbaria, M.; Zinatelli, N.; Cragg, P.; and Cavaye, A. Personal computing acceptance
factors on small firms: A structural equation model. MIS Quarterly. 21. 3 (1997), 279-302.
THE D E L O N E AND Me LEAN MODEL OF tNFORMATION SYSTEM SUCCESS
29
18. Ishman, M. Measuring information system success at the individual level in cross-cultural environments. In E.J. Garrity and G.L. Sanders (eds.). Information Systems Success Measurement. Hershey. PA: Idea Group, 1998, pp. 60-78.
19. Ives. B.; Olsen M.: and Baroudi. J.J. The measurement of user information satisfaction.
Communications of the ACM. 26. K) (1983). 785-793.
20. Jiang, J.J., and Klein, G. User evaluation of information systems: By system typology.
IEEE Transactions on Systems. Man. and Cybernetics, 29. 1 (1999). 111-116.
2L Jiang. J.J.; Klein. G.; and Carr, C.L. Measuring information systems service quality:
SERVQUAL from the other side. MIS Quarterly. 26. 2 (2002), 145-166.
22. Jurison, J. The temporal nature of IS benefits: A longitudinal study. Information & Management, M 2(1996). 75-79.
23. Kaplan, R.S.. and Norton. D.P. Translating Strategy Into Action: The Balanced Scorecard.
Boston: Harvard Business School Press. 1996.
24. Karahanna. E.; Straub, D.W.; and Chervany, N.L. Infomiation technology adoption across
time: A cross-sectional comparison of pre-adoptton and post-adoption beliefs. MIS Quarterly.
2J. 2(1999), 183-213.
25. Kettinger, W.J., and Lee, C.C. Perceived service quality and user satisfaction with the
information services function. Decision Sciences. 25. 5-6 (1995). 737-765.
26. Larsen, T.J., and Wetherbe, J.C. An exploratoiy field study of the ditferences in information technoiogy use betv^-een more- and less-innovaiive middle managers. Information & Management. 36. 2 (1999). 93-108.
27. Lassila, K.S., and Brancheau. J.C. Adoption and utilization of commercial soltware packages: Exploring utilization equilibria, transitions, triggers, and tracks. Journal of Management
Information Systems. 16. 2 (Fall 1999), 63-90.
28. Li, E.Y. Perceived importance of infonnation system success factors: A meta analysis of
group differenceii. Information & Management, 32, I (1997). 15-28.
29. Liu, C , and AmetL K.P. Exploring the factors associated with Web site success in the
context ol electronic commerce. Information and Managetnent, 38. 1 (2(K)0). 23-33.
30. Martinsons. M.; Davison, M.R.; and Tsc. D. The balanced scorecard: A foundation for the
strategic management of information systems. Decision Support Systems, 25, 1 (1999), 71 -88.
31 .Mason, R.O. Measuring information output: A communication systems approach. Information & Management. I. 5 (1978). 219-234.
32. Mirani, R., and Lederer. A.L. An instrument for assessing the organizational benefits of
IS projects. Decision Sciences. 29. 4 (1998). 803-838.
33. Molla, A., and Licker. P.S. E-commerce systems success: An attempt lo extend and
respecify the DeLone and McLean model of IS success. Journal of Electronic Commerce Succc.v.v, 2. 4(2001). 1-11.
34. Myers. B.L.; Kappelman. L.A.; and Prybutok. V.R. A comprehensive mode! for assessing the quality and productivity of the information systems function: Toward a theory for
information systems assessment. In E.J. Garrity and G.L. Sanders (eds.). Information Systems
Success Measurement. Hershey. PA: Idea Group, 1998. pp. 94-121.
35. Newman. M.. and Robey. D. A social process model of user-analyst relationships. MIS
Quarterly. 16. 2 (t992), 249-266.
36. Palmer. J.W. Web site usability, design, and perfomiance metrics. Infonnation Systems
Re.search, 13. 2 (2002). 151-167.
37. Pitt. L.F.; Watson, R.T.; and Kavan. C.B. Service quality: A tiieasure of infomiation
systems effectiveness. MIS Quarterly 19. 2 (1995). 173-188.
38. Rai, A.; Lang, S.S.: and Welker. R.B. Assessing the validity of IS success models: An
empirical test and theoretical analysis. Information Systems Research. 13. 1 (2002). 50-69.
39. Saarinen, T. An expanded instrument for evaluating information systems success. Information & Management. 31, 2 (1996). 103-118.
40. Seddon, P.B. A respecification and extension of the DeLone and McLean model of IS
success. Information Systems Research. S. 3 (1997), 240-253.
41. Seddon. PB., and Kiew. M.-Y. A piu^ial test and development of the DeLone and McLean
model of IS success. In J.I. DeGross, S.L. Huff, and M.C. Munro (eds.). Proceedings of the
International Conference on Information Systems. Atlanta, GA: Association for Information
Systems, 1994. pp. 99-110.
30
DELONE AND MCLEAN
42. Seddon, P.B.; Staples, D.S.; Patnayakuni, R.; and Bowtell. M.J. The dimensions of information systems success. Communications of the A.ssociation for Information Systems. 2. 20
(November 1999) (available al cais.isworld.org/articles/2-20/default a.sp?View=ndf&x=
12&y=13).
^
43. Shannon, C.E.. and Weaver, W. The Mathematical Theory of Communication. Urbana,
IL: University of Illinois Press, 1949.
44. Shirani. A.; Aiken, M.; and Reithel. B. A model of user information satisfaction Data
Base, 25, 4 (\994) 17-23.
45. Smithson. S., and Hirschheim, R. Analysing information system evaluation: Another
look at an old problem. European Journal of Information Systems, 7. 3 (1998). 158-174.
46. Straub. D.; Limayem, M.; and Karahanna-Evaristo, E. Measuring system usage: Implii:auons for ]S theory iesiing. Management Science. 41. 8(1995). 1328-1342.
47. Taylor. S., and Todd. P, Understanding information technology usage: A test of competing models. Information Systems Research. 6. 2 (1995). 144-176.
48. Teng, J.. and Calhoun. K. Organizational computing as a facilitator of operational and
managerial decision making: An exploratory study of managers' perceptions. Decision Sciences, 27, 4 (1996). 673-710.
49. Teo, T.S.H.. and Choo, W.Y. Assessing ihe impact of using the inEemet for competitive
intelligence. Information <fe Management. 39. I (2001). 67-83.
50. Teo. T.S.H., and Wong. P.K. An empirical study of the performance impact of computerization in the retail industry. Omf'g«—r/j('//(/£';7i«;/rwfl/./ou/-Ha/«/"Mflm;£.'£-»;p/ir5£7£'nrf' 26 5
(1998). 611-621.
- f t
51. Thong, J.Y.T., and Yap, C. Information systems effectiveness: A u.ser satisfaction approach. Information Processing & Management. 32. 5 (1996). 601-610.
52. Torkzadeh. G., and Doll, W.J. The development of a tool for measuring the perceived
impact of mformation technology on work. Omega—The International Journal of Management Science. 27. 3 (1999), 327-339.
53. Van Dyke, T.P; Kappelman. L.A.; and Prybutok, V.R. Measuring information systems
service quality: Concern.s on the use of the SERVQUAL questionnaire. MIS Quarterly 21 2
(1997). 195-208.
^
y. •
54. Weill. P. and Vitale. M. Assessing the health of an information system portfolio: An
example from process engineering. MIS Quarterly, 23. 4 {1999). 601-624.
_ 55. Whyte, G.; Bytheway, A.; and Edwards, C. Understanding user perceptions of information system success. Journal of Strategic Information Systems. 6, I (1997). 35-68.
56. Wilkin, C . and Hewitt, B. Quality in a respecification of DeLone and McLean'.s IS
success model. In Mehdi Khozrowpour (ed.). Proceedings of 1999 IRMA International Conference Hershey. PA: Idea Group Publishing, 1999. pp. 663^72.
57. Wixom. B.H., and Watson. H.J. An empirical investigation of the factors affecting data
warehousing success. MIS Quarterly. 25, I (2001), 17-41.
58. Yoon. Y.; Guimaraes, T.; and Cleven.son, A. Exploring expert systems success factors for
business process reengineering. Journal of Engineering and Technology Management 15 2/3
(1998), 179-199.
^
. .
59. Young, D.. and Benamati, J. Differences in public Web sites: The current state of large
U.S. firms. Journal of Electronic Commerce Research, 1, 3 (2000) (available at www.csulb.edu/
web/journals/jecr/p_i.htm).
60. Yuthas, K., and Young. S.T. Material matters: Assessing the effectiveness of materials
management IS. Information <& Management. 33. 3 (1998). 115-124.