International Journal of Accounting Information Systems

8 (2007) 240 – 263

An empirical examination of CobiT as an internal control

framework for information technology
Brad Tuttle ⁎, Scott D. Vandervelde 1
University of South Carolina, Moore School of Business, 1705 College Street, Columbia, SC 29208, USA
Received 26 September 2006; received in revised form 25 September 2007; accepted 30 September 2007

Abstract

One commonly used framework for developing and evaluating technology intensive information
systems is CobiT. This framework was originally a benchmark of best control practices developed and
maintained by the Information Technology Governance Institute, the umbrella organization to the
Information Systems Audit and Control Association. We empirically examine the conceptual model that
underlies the CobiT internal control framework as it applies to an audit setting (including operational,
compliance, and financial audit settings). We find that superimposing CobiT's conceptual model onto audit
relevant assessments made by a panel of highly experienced IT auditors confirms the internal consistency
between the underlying constructs of CobiT. Furthermore, we find that CobiT's conceptual model predicts
auditor behavior in the field related to their seeking help and giving help as evidenced by their postings to a
general IT audit listserv. Given the results of this study, we propose future research aimed at developing a
general theory of internal control applicable to information technology based on CobiT.
© 2007 Elsevier Inc. All rights reserved.

Keywords: Internal controls; IT controls; Internal control frameworks; CobiT

1. Introduction

Organizations and their auditors use frameworks to guide their design and evaluation of internal
controls. This use of internal control frameworks has dramatically increased in importance since
the passage of the Sarbanes–Oxley Act of 2002 and since the release of the Public Company
Accounting Oversight Board's (PCAOB) Auditing Standard No. 2 (AS2) in 2004. Presumably, the

⁎ Corresponding author. Tel.: +1 803 777 6639; fax: +1 803 777 0712.
E-mail addresses: tuttle@moore.sc.edu (B. Tuttle), vandervelde@moore.sc.edu (S.D. Vandervelde).
1
Tel.: +1 803 777 6075; fax: +1 803 777 0712.

1467-0895/$ - see front matter © 2007 Elsevier Inc. All rights reserved.
doi:10.1016/j.accinf.2007.09.001
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 241

use of a framework to guide the assessment of internal controls results in more comprehensive,
reliable, and complete assessments. To achieve these goals in today's information technology (IT)
intensive environment, a control framework must conceptualize the important aspects of internal
control within an IT context in a complete and logically consistent manner. In the absence of a
comprehensive and conceptually sound framework, the complexity of modern systems can
overwhelm an auditor. This suggests that the quality of the internal control audit assessment
depends on the conceptual model upon which a framework rests.
This paper looks at Control Objectives for Information Related Technology (CobiT) by
examining its conceptual consistency in an audit setting. Within IT intensive environments, CobiT
is a widely recognized control framework that is emerging as the supplemental framework of
choice to the Treadway Commission's Committee of Sponsoring Organizations (COSO) eval-
uation framework (IT Governance Institute, 2005; see also Colbert and Bowen, 1996; Netegrity,
2004; Ramos, 2004). Fedorowicz and Gelinas (1998) state that CobiT complements the COSO
framework for assessing the internal controls and overall corporate governance of an organization.
Likewise, Lainhart (2001, 19–20) states that CobiT is a tool that “helps enterprises balance IT risk
and investment in controls.” These sentiments are echoed by Dennis Reynolds, KPMG partner and
head of the Financial Services Risk Governance in London (KPMG, 2003 13),
The Committee of Sponsoring Organizations of the Treadway Committee (COSO) evaluation
framework is recommended by the Commission as an appropriate basis for management's
assessments. Most international organizations are adopting the COSO framework for their
evaluation, but are supplementing its control criteria with those recommended by the Control
Objectives for Information and related Technology (CobiT)…2
CobiT was originally intended for use by an organization's management as a benchmarking
tool consisting of the best practices related to IT controls. Since then and because of its strong
control focus, both internal and external auditors have applied CobiT to financial statement audits
as well as to operational and compliance audits. In regards to financial statement audits, AS2
mandates that management use a control framework in order to assess the effectiveness of internal
controls over financial reporting. The use of the term “framework” in this paper is in the same vein
as used in AS2. While CobiT is apparently useful for financial statement audit purposes, this
study takes a broader view to include internal controls related to operational and compliance
audits.
Despite the importance of using a sound conceptual model, no practitioner developed internal
control framework, that we are aware of, has undergone rigorous academic examination in the
same manner that researchers routinely examine the conceptual models developed by other
academics. The objective of the present study is to examine the internal consistency of CobiT's
conceptual model within an audit setting by investigating whether auditor perceptions of audit
risk related to complexity, client importance, client attention, and process risk combine to
represent IT process risk in the manner asserted by CobiT. (See Appendix A for definitions and
Section 3.1 for a discussion of these measures of risk.) The present study provides further

2
Perhaps one measure that CobiT is generally regarded as an appropriate supplement to COSO is found in the
following entry to the Wikipedia dictionary (http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act): “The PCAOB suggests
considering the COSO framework in management/auditor assessment of controls. Auditors have also looked to the IT
Governance Institute's “CobiT": Control Objectives of Information and Related Technology for more appropriate
standards of measure. This framework focuses on IT processes while keeping in mind the big picture of COSO's control
activities and information and communication.” The authors did not submit this entry to the dictionary just so we could
include it in our paper.
242 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

collaborating evidence as to whether the extent to which auditors seek and give IT audit assistance
reflects the conceptual model underlying the CobiT framework.
An examination of the consistency of CobiT's conceptual model is important for three reasons.
First, users of CobiT as a framework in order to maintain effective IT control would benefit from
knowing if the underlying conceptual model holds together under scrutiny. In this manner,
internal audit functions can use CobiT with increased confidence as a framework for any type of
IT audit they perform, whether it is an operational audit, compliance audit, or financial audit.
Second, based on discussions with practicing auditors, the major public accounting firms use
either CobiT, or something very similar, when working on clients with significant IT controls.
Policy-makers would benefit from evidence that either supports or calls into question the
conceptual foundations of current audit practice. Having this data would greatly aid policy-
makers who must set auditing standards related to internal controls for publicly traded companies.
Third, it is possible that examining CobiT's conceptual model is a first step toward the devel-
opment of a more general theory of internal control. Although no academic theory of internal
control exists, the profession is essentially proposing CobiT as a process oriented theory of
internal control based on IT processes, IT domains, information criteria and the IT resources
employed to generate information. Further development of a formal theory of internal control,
especially as it relates to IT, should lead to more effective compliance and operational audits.
This study uses audit related assessments of CobiT constructs provided by a panel of experts
together with postings from the Information Systems Audit and Control Association's (ISACA)
general listserv to provide empirical support for CobiT as an IT internal control framework to
support the audit function (including operational, compliance, and financial audits). First, a panel
of 12 highly experienced IT auditors evaluated key aspects of CobiT's conceptual model using the
following measures of audit risk with regards to IT: complexity, client importance, client
attention, and process risk. Analysis shows that responses from the expert panel combine in a
manner that is generally consistent with CobiT's underlying conceptual model. We discuss
departures in our findings from the CobiT model in the final section of the paper. Second, the 12
IT auditors along with 17 non-IT auditors evaluated each of the CobiT processes based on the
level of risk to a typical organization. These ratings correlate with data obtained from the ISACA
listserv. The analysis suggests that the conceptual model upon which the CobiT framework is
based is associated with (1) the number of threads or topics discussed on the ISACA listserv, and
(2) the total number of messages posted. To further support the CobiT conceptual model, all
relevant listserv messages were successfully coded to specific CobiT control objectives based on
their content. That is, CobiT appears to be sufficiently comprehensive to encompass the audit
specific questions posted in our sample from the listserv.
The remainder of the paper proceeds as follows. Section 2 provides the background, theoretical
framework, and research propositions. Section 3 provides the first phase of the CobiT validation
based on experienced auditors. Section 4 provides the second phase of the CobiT validation using
archival information. Finally, Section 5 provides a discussion of our results.

2. Background and theory

The well-established COSO framework relies on the idea that the achievement of the following
objectives is important for strong internal controls: effectiveness and efficiency of operations,
reliability of financial reporting, and compliance with applicable laws and regulations (COSO,
2004). Its underlying conceptual model suggests that internal control objectives are achieved by
paying attention to five components of control: (1) the control environment, (2) risk assessment,
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 243

(3) control activities, (4) information and communication, and (5) monitoring. By conceptualizing
internal controls in this manner, control frameworks are meant to accomplish three objectives:
(1) ensure completeness in coverage, (2) aid in identifying high risk areas, and (3) help to
accurately assess the impact of controls (COSO, 1992). As a practical matter, however, COSO is a
highly abstract conceptual framework and does not identify control objectives at a level of
specificity sufficient to design detailed audit tests. Furthermore, the general nature of COSO does
not address the complexity and special risks inherent in IT (Colbert and Bowen, 1996). Given the
reliance on technology within most organizations, organizations need a framework to address
technology to be functional in today's audit environment. Furthermore, because COSO expresses
its components at a very high level of abstraction, it may not be possible to design an empirical
test of its internal conceptual consistency.
For these reasons, organizations and auditors in computerized environments are adopting
specialized frameworks, such as CobiT, to supplement COSO. Every major international
accounting firm has adopted CobiT or at a minimum its major constructs in connection with their
review of internal control. This trend extends beyond the U.S. as evidenced by the European
Union's recent adoption of CobiT as an Auditing Standard (Summerfield, 2005). Unlike COSO's
five components, which are structured by semantic category, the CobiT framework relies on a
process model that is organized around a system life cycle approach containing four primary
domains (see Fig. 1). These domains are labeled: Plan and Organise; Acquire and Implement;
Deliver and Support; and Monitor and Evaluate. Within each domain there are specific processes
that an organization should address to achieve detailed and specific IT related control objectives.
For instance, within the Deliver and Support domain is the process, “DS4 Ensure Continuous
Service.” This process is associated with 10 detailed control objectives (not listed in the figure)
that IT best practices suggest should be met in order to achieve a high level of control. An
example control objective from this process is objective DS4.6, “IT Continuity Plan Training”
which states, “Ensure that all concerned parties receive regular training sessions regarding the
procedures and their roles and responsibilities in case of an incident or disaster. Verify and
enhance training according to the results of the contingency tests” (IT Governance Institute, 2005,
116). These detailed control objectives are further supplemented by audit guidelines for each
CobiT process. It is important to note that the control objectives in CobiT are specific enough to
be easily implementable; yet general enough to be applicable to various types of audits (e.g.,
operational, compliance, and financial).
CobiT's underlying conceptual model asserts that to satisfy business requirements, information
must meet seven criteria: (1) Effectiveness, (2) Efficiency, (3) Confidentiality, (4) Integrity,
(5) Availability, (6) Compliance, (7) and Reliability (Appendix B provides detailed descriptions
for each criterion as presented in CobiT 4.0). The conceptual model relates each CobiT process to
the information criteria that the process affects, and therefore, should provide an auditor with a
means of directly assessing specific controls for their effect on the quality of information, whether
the audit is operational, compliance, or financial in nature (see Fig. 1). Furthermore, there are clear
linkages between the CobiT information criteria and COSO's objectives related to effectiveness
and efficiency of operations, compliance with laws and regulations, and reliability of information.3
Achieving the CobiT information criteria, therefore, has important implications for financial
statement assertions as well as broader implications for the efficiency and effectiveness of
operations.

3
See CobiT 4.0 Appendix II, “Mapping IT Processes To IT Governance Focus Areas, COSO, CobiT IT Resources And
CobiT Information Criteria” (IT Governance Institute 2005).
244 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Fig. 1. CobiT Version 4.0 conceptual model.

 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 245

For each CobiT process, the IT resources (i.e., assets) that the process affects are also identified.
These resources consist of (1) People, (2) Information or data, (3) Applications, and
(4) Infrastructure.4 Melville et al. (2004) provide a related business valuation model based on
existing research that shows the important components of a good IT structure. The primary
components of Melville et al.'s model consist of physical capital, human capital, and
organizational capital. We note that each component of Melville et al.'s model maps directly
into at least one of the IT resources within the CobiT framework. CobiT expands this model by
adding Information as a critical IT resource. As is identified by Melville et al. (2004), research has
shown that strong IT improves organizational performance.
Given the importance of IT to organizational performance and the direct link between the
primary components of the business value model and the IT resources of CobiT, one can expect
risk assessments for a particular process to correlate with the IT resources it affects. For example,
if CobiT's conceptual model is internally consistent and the IT resource, people, is considered a
relatively high risk factor, then the assessed risk of not satisfying a particular CobiT process that is
closely associated with the people resource should reflect this heightened risk. Likewise, a test of
internal consistency within an audit context involves examining whether risk assessments for the
various CobiT processes correlate with those information criteria which the conceptual model
purports the process to affect. That is, suppose the information criterion of integrity is more
complex, difficult to audit, or otherwise risky in comparison to the other information criteria. In
this case, the risk of not satisfying a CobiT process associated with the integrity criterion should
reflect its underlying complexity, etc. These arguments suggest that if CobiT is internally
consistent, then the information criteria and IT resource constructs should exhibit construct
validity. This leads to our first proposition:
Proposition 1. Risk assessments for CobiT processes will be correlated with the underlying audit
related characteristics for the information criteria and IT resources CobiT purports that the
process affects.

An additional proposition is set forth in relation to the propensity of auditors to seek additional
information and assistance via postings to the Information Systems Audit and Control Association
(ISACA) IT audit listserv.5 This particular listserv consists of audit related questions posed by IT
auditors to other IT auditors and is distinct from a separate listserv maintained by ISACA that
addresses specific CobiT issues.6 Hence, message threads on the listserv provide an unobtrusive
measure of the need to obtain assistance and information on specific topics within a broad IT audit
setting. Furthermore, the total number of postings (i.e., questions and responses) provides a measure
of what it takes to provide a satisfactory answer. The number of postings related to a specific topic is a
proxy for the complexity or the importance of the topic to auditors. The more complex the issue, the
less likely that one response to a posted question will be sufficient to provide a complete solution, and
will therefore lead to more posted responses and follow-up questions. The more important the topic,
the more likely multiple people will respond to the posted message with a response. Therefore, a
strong test of the internal consistency of CobiT's underlying conceptual model is to examine its
association with the frequency with which auditors seek assistance via the listserv.

4
CobiT 3.0 identified five IT resources. CobiT 4.0, which was released in November of 2005, combined the
Technology and Facilities IT resources into Infrastructure, thus reducing to four IT resources.
5
See www.isaca.org.
6
ISACA maintains several specialized listservs. Using the general audit listserv provides a stronger test of CobiT's
applicability to an audit setting than using the listserv dedicated solely to CobiT issues.
246 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Data from the listserv provides a stronger test of the CobiT model's internal consistency than
would data about purposeful auditor behavior, such as data about audit plans and tests. This is
because auditors might adopt and use an internally inconsistent conceptual model based on their
training and culture without being aware of the model's shortcomings. Seeking and giving help, on
the other hand, is a spontaneous behavior that is likely to reflect audit needs associated with the risk
associated with the particular issue. For instance, if CobiT constitutes an internally consistent
conceptual model in an audit setting, one expects ITauditors to serendipitously seek help with CobiT
processes when the information criteria associated with that process embody important audit
characteristics related to various aspects of risk, such as its complexity, importance. That is, a
process, which is complex, and that impacts information criteria should create a greater need for
audit assistance. It should elicit, therefore, greater assistance from other auditors. This logic suggests
that the help auditors seek in relation to a CobiT process will be influenced by the information criteria
related to that process. The same expectation exists in regards to IT resources. This leads to our
second proposition:
Proposition 2. CobiT's conceptual model (i.e., processes, information criteria, and IT resources)
predicts the extent to which an auditor will seek and give assistance related to an IT audit topic.

Proposition 2 is a strong test of the theoretical consistency of CobiT's conceptual model

because in order to support its premise the CobiT processes must be valid categories of IT
activities within an audit context.7 Otherwise, if CobiT processes are not audit relevant then no
association between CobiT and the need for information and assistance in an audit context (i.e.,
listserv postings) will result. Furthermore, CobiT's assertions regarding which information
criteria and which IT resources are affected by each process are subject to the same logic. If CobiT
does not extend to an audit context then no association will result between the amount of
information an auditor seeks and the need for information related to the listserv topic as implied
by the information criteria and IT resources asserted by the CobiT framework.
In addition to the information criteria and IT resource taxonomy, beginning with Version 4.0,
CobiT identifies the importance level it places on each CobiT process as shown in Fig. 1. If the
CobiT framework is applicable to an audit setting, then the importance level (i.e., H = high,
M = medium, or L = low) should be useful in audit planning and should correlate with the
propensity of auditors to seek and give advice on the listserv. We, therefore, present our third and
final proposition:
Proposition 3. There is a positive association between the perceived importance level of an IT
audit topic and the extent to which auditors seek assistance related to that particular ITaudit topic.

3. Test of CobiT using responses from experienced auditors

3.1. IT and non-IT auditor assessments

As a first step to test our propositions regarding CobiT's conceptual model, audit relevant
assessments of CobiT constructs were obtained from an expert panel consisting of 12 highly
experienced IT auditors and 17 non-IT auditors.8 The IT auditors have a mean full-time work
7
Regardless of the results of this study, CobiT may still be a valid framework for IT related purposes other than as an
internal control framework in an audit setting.
8
Of the 17 non-IT auditors one did not complete the demographic questions of the questionnaire, so the information for
only 16 is reported here.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 247

experience of approximately 11.9 years, whereas the non-IT auditors have a mean work experience
of approximately 2.1 years. Seven IT auditors held four-year college degrees and six held graduate
degrees. Sixteen non-IT auditors held graduate degrees. Eight of the IT auditors work for a
“professional assurance/consulting firm,” while four work in “insurance, real estate, finance,
banking, and accounting.” Fifteen of the non-IT auditors work for a “professional assurance/
consulting firm,” while one works in “insurance, real estate, finance, banking and accounting.” The
professional designations possessed by our IT auditors include three CPAs, eight CISAs, three
CIAs, one CCNA, one CFE, one CFSA, and one CISSP. Two panel members did not indicate a
professional designation. The non-IT auditors include eight CPAs. The mean self-assessed
knowledge of the IT concepts in the questionnaire for the IT auditor and non-IT auditor groups
respectively is 6.6 and 6.0 on a nine-point scale with one being “low knowledge of the concepts”
and nine being “high knowledge of the concepts.” The mean self-assessed familiarity with CobiT is
7.0 and 4.6 on a nine-point scale with one being “very unfamiliar with CobiT” and nine being “very
familiar with CobiT,” for the IT auditor and non-IT auditor groups, respectively. The non-IT
auditors responded (mean = 3.1) to the question, “Please indicate the extent to which you
personally apply CobiT in your day-to-day work activities” on a nine-point scale with one being
“Very infrequently” to nine being “Very frequently.” These results suggest that the IT auditors have
significant familiarity with CobiT and that even the non-IT auditors have some familiarity.
The 12 highly experienced IT auditors performed three tasks in which they evaluated the
following: (1) each of the information criteria, (2) each of the IT resources, and (3) assessed the
risk associated with each of the 34 CobiT processes.9 The evaluation tasks took approximately
40 min to complete. The 17 non-IT auditors assessed only the risk associated with each of the 34
CobiT processes (i.e., task 3).
The evaluation of each of the information criteria and each of the IT resources was based on
providing evaluations for four measures of audit risk: complexity, client importance, client
attention, and process risk.10 Consistent with the desirability of obtaining multiple measures of a
single construct, each of these evaluations is expected to capture slightly different aspects of audit
risk in an IT setting so that when combined, a better overall measure of IT audit risk emerges. Our
expectation is that an auditor will seek more information about a CobiT process associated with
an information criterion or IT resource that is complex, important, requires more attention or high
risk. The motivation for each measure appears below.

3.1.1. Complexity
Complexity in a process or transaction increases the risk of material misstatement, and
therefore, increases audit risk. As an example, in reference to revenue recognition, in Messier,
Glover, and Prawitt's 5th edition auditing textbook they state “recognition of revenue may
involve complex calculations….In such circumstances, the auditor should assess the risk of
material misstatement to be high” (2008, p. 386). As pointed out by Ridley et al. (2004, p. 7) the
more complex the IT governance of an organization, “it is likely that there will be more interest
in IT control from these organizations” (Ridley et al., 2004, p. 7). “The challenges include

9
At the time of the expert panel participation, CobiT 3.0 was being used. Therefore, the panel made judgments on five
IT resources. As indicated in note two, CobiT 4.0 combines Technology and Facilities. During the analysis of the expert
panel responses, these two IT resources are combined to reflect CobiT 4.0.
10
Audit difficulty was also measured but is not correlated with the other risk measures and is therefore not included in
any analyses. Initially audit difficulty was included as it seemed reasonable that it would impact auditor risk judgments.
As it turns out, it appears that our participants viewed audit difficulty more as being related to detection risk in the audit
risk model, while the other factors relate to inherent risk and control risk associated with the client's system.
248 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

identifying a hornet's nest of controls and interfaces among decentralized business units and
trying to manage the efforts with scarce resources” (Hoffman, 2004, p. 2).

3.1.2. Importance
As a whole, “the importance of IT governance can be appreciated in light of Gartner Group's
finding that large organizations spend over 50% of their capital investment on IT” (Ridley et al.,
2004, p. 1). More specifically, we expect management to place more importance on IT areas that
are high risk. For instance, self-assessment programs recognize the relationship between risk and,
“how important the process is for their business objectives” (Lainhart, 2001, p. 21). The more
important the IT process, the more likely auditors are to seek assistance when a question arises.

3.1.3. Attention
Similar to importance, we argue that organizations focus their attention with respect to IT
governance on areas of higher risk. This notion is consistent with self-assessment programs and
the risk-based audit approach applied by major accounting firms. “Management can concentrate
on the areas of high risk identified through auditors' assessments and then use CoBiT's high level
and detailed control objectives to determine cost-effective means for mitigating these risks”
(Lainhart, 2001, pp. 21–22).

3.1.4. Process risk

The risk associated with a process failing, regardless of the “type” of risk (i.e., inherent risk,
control risk, business risk, or fraud risk) increases audit risk. “COBIT is a breakthrough tool that
helps enterprises balance IT risk and investment in controls” (Lainhart, 2001, pp. 19–20).
Each evaluation was elicited on a scale from one to nine with one being “very low [insert one
of the four measures]” and nine being “very high [insert one of the four measures]” (i.e., “very low
complexity” and “very high complexity”) as it relates to a “typical organization.” Table 1 shows
mean ratings from the IT auditors for each information criteria and IT resource for each measure.
Although the IT auditors expressed a high level of familiarity with the concepts in the survey and
with CobiT, to ensure reliability the IT auditors first read the CobiT definitions of the information
criteria and IT resources before completing their assessments (IT Governance Institute, 2000).11
We perform an exploratory factor analysis to determine whether the above measures load on a
single risk factor. This analysis suggests that complexity, client importance, client attention, and
process risk load on the same factor.
Table 1 also shows what we label a “Combined Assessment,” for each information criterion
and each IT resource. The combined assessment represents an “overall” risk measure score that is
derived by weighting the assessments for each of the four different risk measures (complexity,
client importance, client attention, and process risk) by their factor scores and then summing the
products (this process is described in more detail below). The combined assessments are used to

11
The manner in which we obtain the ratings makes it highly likely that the responses result from knowledge gained
from experience rather than from training in CobiT. For instance, Fig. 1 of the paper shows that there are 226 instances in
which an Information Criteria or IT Resource apply to one of the 34 CobiT processes. In addition, three importance levels
are associated with 34 separate IT processes. Both facets of the framework far exceed a person's ability to recall from
memory based on training alone. More importantly, it is unlikely that our auditors previously encountered ratings of risk,
complexity, client attention, etc. as they apply to the specific Information Criteria or IT Resources or CobiT processes. In
contrast, the IT auditors do encounter circumstances during their audits in which they learn to associate these factors
together.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 249

Table 1
Expert panel assessments of CobiT information criteria and IT resources
Mean audit related assessments
Framework construct Combined assessment Complexity Client Client Process
(Std Dev) importance attention risk
Factor scores (Weight) a 0.79560 0.63194 0.90126 0.94105

Panel A: Information criteria

Effectiveness 5.34 6.42 7.25 6.17 6.50
Efficiency 4.51 6.50 6.25 4.58 5.08
Confidentiality 5.57 6.25 6.67 6.25 7.92
Integrity 5.70 6.67 7.33 6.25 7.67
Availability 5.23 5.25 7.50 6.25 6.75
Compliance 5.98 6.33 7.50 7.17 8.17
Reliability 5.72 5.58 7.92 6.83 7.75

Panel B: IT resources
People 5.38 5.67 7.33 6.50 6.92
Information 6.21 6.75 8.33 7.50 7.92
Applications 6.27 7.00 7.92 7.92 7.83
Infrastructure a 5.24 6.04 6.96 6.21 6.54
CobiT 4.0 released in November 2005 combined the IT Resources of technology and facilities into one category, which it
labeled “Infrastructure.” The mean audit related assessments in the table for Infrastructure consists of the expert ratings for
technology and facilities added together, divided by two. Combined assessments equal the sum of the audit related
assessments weighted by the factor scores.
a
The prior communality estimate for each variable was set at its squared multiple correlation with all other variables.

create a “framework” score according to the relationships inherent in CobiT. The framework
scores are then used for predicting auditor listserv behavior as described later.
Regarding the third task, all auditors (the 12 IT auditors and 17 non-IT auditors) were asked to
“consider the risk to the typical organization associated with an unsatisfactory outcome in each of the
following CobiT processes.” They indicated their CobiT process assessments of risk on a scale from
one to nine with one being “very low risk” and nine being “very high risk.” Mean risk assessments
for each CobiT process are shown in Table 2 along with framework scores (explained below).
Framework scores are computed for each CobiT process in the following manner: (1) the
information criteria and IT resources associated with the process are coded as a one if indicated in
Fig. 1 and as a zero otherwise, and (2) the result is multiplied by the combined assessments shown
in the first numeric column of Table 1. These numbers are then summed and averaged into a single
framework score for each CobiT process.12 The framework scores permit a test of whether CobiT's
conceptual model, from an audit relevant standpoint as reflected in the expert panel's assessments
of the information criteria and IT resources, predicts IT auditor behavior in seeking and providing
assistance on IT audit related topics. The use of the framework score allows the analysis to reflect
audit risk as measured serendipitously by the information criteria and IT resources associated with
each CobiT process.
To see how the framework scores are computed, recall that the CobiT conceptual model is
categorized into four domains, which in turn are associated with 34 IT processes. Conceptually,

12
Framework scores consisting of a simple count (i.e., unweighted by the combined assessments of the experts) produce
essentially the same results as analysis using weighted framework scores.
250 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 2
Expert panel: mean risk assessments and framework scores
CobiT process Description N = 29 risk N = 12 framework
assessment scores a
PO1 Define a strategic IT plan 7.24 3.00
PO2 Define the information architecture 6.24 3.05
PO3 Determine technological direction 6.14 1.94
PO4 Define the IT processes, organization and relationships 6.86 1.38
PO5 Manage the IT investment 6.28 2.95
PO6 Communicate management aims and direction 6.59 2.08
PO7 Manage IT human resources 5.83 1.38
PO8 Manage quality (PO 11 from CobiT 3rd ed.) 7.21 4.03
PO9 Assess and manage IT risks 7.72 5.56
PO10 Manage projects 6.62 2.43
AI1 Identify automated solutions 6.83 1.94
AI2 Acquire and maintain application software 6.86 2.50
AI3 Acquire and maintain technology infrastructure 6.59 2.36
AI4 Enable operations and use 7.14 4.49
AI6 b Manage changes 8.14 4.51
AI7 Install and accredit solutions and changes (AI 5 from CobiT 3rd ed.) 7.41 4.51
DS1 Define and manage service levels 6.21 5.56
DS2 Manage third-party services 6.59 5.56
DS3 Manage performance and capacity 6.62 2.42
DS4 Ensure continuous service 6.86 3.47
DS5 Ensure systems security 8.00 4.66
DS6 Identify and allocate costs 5.69 3.03
DS7 Educate and train users 6.97 1.38
DS8 Manage service desk and incidents 5.45 1.95
DS9 Manage the configuration 6.90 3.50
DS10 Manage problems 7.34 3.47
DS11 Manage information 7.62 1.60
DS12 Manage the physical environment 6.28 1.47
DS13 Manage operations 6.86 3.99
ME3 c Ensure compliance with external requirements 7.90 3.16
a
Framework Score represents the combined assessments from Table 1 for each information criteria and IT resource
identified to affect the given CobiT process. After the respective combined assessments are added together, the total is
divided by 11 to arrive at the framework score.
b
The assessments made by the expert panel were based on the CobiT 3.0 framework. Cobit 4.0 released in November
2005 has added a new AI5. The previous AI5 under CobiT 3.0 is now AI7 and is shown as such in this table.
c
The CobiT process was PO8 in CobiT 3.0. The remaining Monitor and Evaluate processes in CobiT 3.0 do not map to
CobiT 4.0, and therefore, are not used in the analysis.

CobiT asserts that each process affects two or more information criteria and one or more IT
resource as shown in Fig. 1. If the information criterion has a dot in the row for a given process,
the process is said to affect the information criterion. Likewise, if the IT resource has a dot in the
row for a given process, the process is said to affect the IT resource. For example, Fig. 1 shows
that the process “PO3 Determine Technological Direction” affects the effectiveness and efficiency
information criteria and the applications and infrastructure IT resources. The combined
assessments (from Table 1) for the effectiveness (5.34) and efficiency (4.51) information criteria
and the applications (6.27) and infrastructure (5.24) IT resources sum to 21.36. Dividing by 11
(seven information criteria + four IT resources) produces a framework score of 1.94 as shown in
Table 2. This score represents the IT auditors' assessment of the complexity, client importance,
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 251

client attention and process risk associated with CobiT process PO3 as reflected by the
information criteria and IT resources that are identified as applying to this particular CobiT
process. From a practical standpoint, one can think of a higher score as representing what CobiT's
conceptual model asserts should be an increased audit risk for that particular CobiT process.

3.2. Preliminary analysis

As a first step to testing the internal consistency and internal validity of CobiT, we perform
confirmatory factor analysis to verify that the combined assessments from Table 1 for the seven
information criteria and the four IT resources fall into these two categories as asserted by CobiT.
The resulting model fails to fit the data and so we next conduct an exploratory factor analysis
(with varimax rotation) to see what factors might underlie the IT auditors' assessments. As shown
in Table 3, the rotated factor pattern produces three factors. Together, these factors account for
84.5% of the variance. The Reliability, Confidentiality, Integrity, and Efficiency information
criteria load on the first factor. The Compliance and Effectiveness information criteria and the
People and Data IT resources load on the second factor. The Availability information criteria and
the Applications and Infrastructure IT Resources load on the third factor. This analysis suggests
that information criteria and IT resources represent three separate constructs that do not conform
strictly to the information criteria and the IT resources categories presented in CobiT. We discuss
the implications of this finding in the discussion section of the paper. Further analysis is not
dependent on these measures loading on their respective CobiT categories.

3.3. Proposition 1—CobiT internal consistency and risk assessments

Proposition 1 provides an initial test of CobiT's internal consistency by examining whether

expert panel assessments related to the information criteria and the IT resources combine, as
asserted by CobiT's conceptual model, in a manner that they correlate with separate risk
assessments for the CobiT process. To test this proposition the risk assessments for the CobiT
processes from Table 2 are regressed on the framework scores from the same table using a
generalized linear model while controlling for individual differences among the expert IT
auditors. The results shown in Table 4 indicates that the framework scores are highly associated

Table 3
Rotated factor pattern based on expert panel combined assessments

IC = Information Criteria, R = IT Resources.

252 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 4
GLM analysis of expert risk assessments for CobiT processes regressed on framework scores from Table 2
Variable df Type III sum of squares F-Value p-value
Auditor 11 88.0043 4.02 b0.0001
Framework 1 59.5932 29.94 b0.0001
Model 12 168.3678 7.05 b0.0001
Error 311 618.9994
R2 = 0.2138
Each expert is associated with a vector containing one row for each CobiT process and two variables in each row: a
framework score and the dependent measure, i.e., the expert's risk assessment for the process (see Table 2 for the
framework score and risk assessment). Framework is computed by first coding each Information Criterion and IT Resource
as a zero or one variable per Fig. 1. These are then multiplied by the combined assessments from Table 1 to arrive at
weights for each Information Criterion and IT resource based on the dimensions of (1) complexity, (2) client importance,
(3) client attention, and (4) process risk. The resulting numbers are then summed for each CobiT process to arrive at the
framework scores found in Table 2 and used in this analysis.

with the IT auditor's risk assessments (F = 29.94; p b 0.0001).13 Note that the risk assessment used
in the framework score is based on the IT experts' assessment of risk associated with each
information criterion and IT resource, and is combined with the other audit relevant assessments
per the CobiT conceptual model. In contrast, the risk assessment used here as the dependent
variable is provided directly from each of the IT auditors' for each of the 34 CobiT processes.
Hence, using two distinct measures, one that focuses solely on the CobiT processes (risk
assessments) and the other that focuses solely on the information criteria and IT resources
(framework scores), the CobiT conceptual model demonstrates significant (internal) convergence
and consistency, thus supporting Proposition 1. The significant effect of auditor simply controls
for individual differences between auditors in overall risk assessments.

4. CobiT examination using archival information

The primary examination of the CobiT conceptual model involves analyzing postings to a
general IT audit listserv maintained by ISACA in which auditors post and respond to IT audit
related questions. The ISACA listserv provides a surrogate for how IT processes may affect audit
risk in that audit risk should influence auditor behavior with respect to seeking and giving audit
help.14 Auditors are more likely to seek help for something they audit that is of higher risk, and
they are more likely to audit IT processes that they associate with more risk. Furthermore, auditors
are more likely to question their own abilities and knowledge and thusly be more likely to ask for
help on issues of importance and high complexity (i.e., high risk). Auditors are also more likely to
take their time to respond to questions posted on a listserv when they believe the question is
important, complex, or related to a highly risky issue. For these reasons, we argue that a listserv in
which IT auditors seek and give audit related advice to each other is an unobtrusive measure of
audit risk that can be categorized as to the topic being discussed.

13
Only the IT auditors are used in this analysis, because they are the only participants who provided assessments of the
information criteria and the IT resources that make up the framework scores. The significant result on the auditor variable
(coded as a class variable) suggests that the assessment of audit risk is significantly impacted by characteristics specific to
the individual auditor.
14
At this time, no publicly available empirical data exists that relates specific IT processes to audit risk. Such data are
unlikely to exist for sometime.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 253

Because the study seeks to examine IT processes in an audit context, any listserv not used by
auditors to discuss IT issues is irrelevant to the current study. At the same time, it is important that
the listserv not bias the findings by specifically addressing only CobiT questions. While we
expect most participants on the general ISACA listserv to be familiar with CobiT, if they had a
question that they believed was specific to CobiT, we expect them to use the CobiT listserv. A
word search on the listserv postings confirms that only six threads contain “Cobit” (not case
sensitive) in the subject line. Hence, the general ISACA listserv appears to address IT audit related
issues by IT audit specialists but does not address issues explicitly arising from the use of CobiT.
Additionally, we gathered risk ratings for the 34 IT processes from 17 non-IT auditors. The
correlation between the 12 IT auditors and the 17 non-IT auditors is 0.718. In general, the IT and
non-IT auditors are similar in their risk assessments on the 34 IT processes.
As shown in Table 5, 616 messages that were posted from January 2002 through December 2003,
and 601 messages that were posted from September 2004 through December 2005 are analyzed.15
These postings related to 342 and 297 separately identified message threads, respectively, in which a
query along with its follow-up responses constitutes a single thread.16 Two Ph.D. students whose
area of study is information systems and who were blind to the purpose of the study separately coded
the message threads, classifying each thread to the CobiT process to which they felt it primarily
related. Their initial agreement rate is 62% (Kappa = 0.59) for the period ending in 2003 and 64%
(Kappa = 61) for the period ending in 2005 which is substantially greater than chance (i.e., 1 out of 34
processes = 3%).17 All disagreements were resolved between the two coders by discussion.
As shown in Table 6, 205 auditors posted to the listserv during the 2002–2003 period, while 152
auditors posted to the listserv during the 2004–2005 period for a total of 357 different IT auditors.
Slightly fewer than half of the auditors posted multiple messages. Three auditors posted more than
30 messages. E-mail addresses were examined for the type of domain and country of origin shown
in Table 7. Although ISACA is an international organization, as can be seen in Table 7, 280
contributors are domiciled in the U.S. (79%), while 74 (21%) are from domains outside the U.S.
CobiT is asserted to be a comprehensive, open standard based on industry best practices, and on
this basis, meets the requirements to be a useful framework. Some indication of the com-
prehensiveness of CobiT is the number of audit specific issues that were posted on ISACA's general
listserv that could not be coded as pertaining to a specific CobiT process by our coders. As can be
seen from Table 5, 21 of the 342 message threads (6.1%) in the 2002–2003 period were not coded to
a specific CobiT process, while 51 of the 296 message threads (17.2%) in the 2004–2005 period
were not coded to a specific CobiT process. Initially, the proportion not coded in the 2004–2005

15
We begin the second time period (September 2004) six months after the issuance of AS2 by the PCAOB (2004) so
that the auditors had time to implement this standard in their work.
16
Although internal controls have been an important component of the audit process, given that the messages from the
first period were immediately surrounding the period when SOX was implemented (and prior to the ratification of AS2),
it is possible that the issues related to internal controls might not be reflected in the postings on the listserv. The second
period is analyzed in order to test to see if there is a delay in the impact of the new regulations on the postings on the
listserv.
17
While both of the PhD students are accounting information systems minors in their doctoral program, one of the
coders had significant audit experience and limited IT experience, while the other coder had significant IT experience and
limited audit experience. Hence, a somewhat higher than normal initial disagreement is expected, but the result of their
combined experience should be a higher quality coding after discussion. However, because of the subjectivity of the
coding process (i.e., a thread may relate to two processes but the coders may disagree on which is the primary process),
we analyze instances in which the coders initially indicated a secondary process to determine if substituting the rejected
classification alters our conclusions. Out of the 639 threads and 1217 messages analyzed in the study, 55 threads and 86
messages are associated with a secondary classification. This re-analysis does not change the interpretation of our results.
254 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 5
Listserv postings
2002–2003 2004–2005
a
CobiT process Thread count Message count Thread count Message count
DS5 114 242 99 196
ME2 113 189 62 138
DS11 25 25 24 67
ME3 13 18 12 16
DS4 9 24 7 18
ME4 6 11 6 10
AI3 4 13 0 0
AI6 4 15 1 3
DS10 4 6 1 1
DS7 4 4 0 0
PO2 4 4 0 0
AI7 3 3 0 0
DS12 3 3 3 3
ME1 3 6 1 2
AI2 2 2 2 4
DS8 2 2 1 8
DS9 2 3 1 1
PO6 2 18 1 2
PO7 2 3 0 0
PO3 1 1 0 0
PO4 1 1 8 23
PO8 0 0 4 12
PO9 0 0 7 7
PO10 0 0 0 0
DS13 0 0 0 0
PO1 0 0 0 0
DS3 0 0 0 0
AI4 0 0 1 3
PO5 0 0 4 12
AI1 0 0 0 0
DS1 0 0 0 0
DS2 0 0 1 1
DS6 0 0 0 0
Not coded 21 23 51 74
Total 342 616 297 601
a
A thread represents a stream of messages that relate to a particular topic.

Table 6
Number of unique contributors by number of postings
Posting frequency Number of unique contributors
2002–2003 Listserv 2004–2005 Listserv
1 110 83
2–5 77 49
6–10 12 14
11–30 5 4
31+ 1 2
Total 205 152
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 255

Table 7
E-mail domiciles
Country and Number of contributors
e-mail domain
2002–2003 Listserv 2004–2005 Listserv
USA
.com 99 83
.edu a 22 11
.gov 5 3
.mil 2
.net 12 9
.org 14 9
.us 7 4
Total USA 161 119

Non-USA
.ae 2 1
.ar 1 1
.au 7 3
.br 1 1
.ca 9 6
.cl 1
.co 1
.de 1
.ed 1
.es 1
.fr 1
.id 1 1
.il 1 2
.in 3
.jo 1
.ke 1 1
.mil 2
.my 2
.nl 2 1
.nz 1
.pe 1
.pk 1
.qa 1
.se 1 1
.tn 1
.tr 2
.uk 2 3
.za 2 2
Total non-USA 42 32
Total 203 151
a
Postings from the.edu domain are IT auditors and not professors. Two 2003 and one 2005 postings do not have e-mail
addresses.

period appears to be relatively high. Analysis of all 72 threads, however, suggests a few reasons why
postings were not coded to specific CobiT processes. A number of threads dealt with issues that were
so general that they span the entire CobiT spectrum or otherwise did not meet the objectives of the
study. By including these in the analysis, it would muddy the results, as they would indicate all of the
CobiT categories apply. Examples include announcements of conferences, meetings, and general
256 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

CPE training. One typical thread in this category dealt with how to use the “COSO framework to
meet Section 404 requirements.” Such a thread regarding the use of COSO could be coded as
applicable to many CobiT processes. A few threads dealt with issues that while not spanning the
entire framework, related to multiple CobiT processes and so could not easily be coded for purposes
of the present study. Discussions related to fraud dominated these threads. A significant number of
threads dealt with current news events rather than specific audit issues. Examples include discussions
about changes to state and national laws, discussions about Internet voting and messages dealing
with U.S. Governmental Accounting Office (GAO) activities. Given that our study examines
whether CobiT covers IT audit issues, non-audit issues from the listserv are not included or
applicable to the analysis. Another set of messages dealt with listserv administration and with job
seeking advice. It seems logical, given the timing of the adoption of AS2, that the 2004–2005 time
period would have a significant number of messages related to seminars and training. Of the 72
threads that were not coded to a specific CobiT process, not one thread related to what could be
considered an IT process that is missing from the CobiT framework. Nevertheless, based on the
nature of our data, these findings must be considered preliminary. After eliminating the threads that
clearly did not represent specific audit questions, the set of potential threads that could not be coded
to CobiT processes is empty. Our subjective analysis, therefore, leads us to conclude that the CobiT
encompasses every audit specific question posted to the listserv.

4.1. Proposition 2—thread and message counts reflect the CobiT conceptual model

Proposition 2 suggests that the underlying complexity, client attention, importance, and process
risk embodied by the information criteria and IT resources associated with each CobiT process should
influence the amount of aid and assistance ITauditors seek and give on the ISACA listserv. In order to
examine Proposition 2, we regress counts of messages and threads coded to each CobiT process as
shown in Table 5 on the CobiT framework scores from Table 2, while controlling for the time period
(2002–2003 versus 2004–2005).18 The framework score reflects both audit related assessments
across the five dimensions and the role that the information criteria and IT resources play in CobiT.
The expectation is that the number of listserv postings should be positively correlated with framework
scores. A positive coefficient indicates that as the number of information criteria and IT resources
increases, and the higher the combined assessment across the four audit risk measures, there will be
more listserv postings. We include time period to test whether implementation experience regarding
SOX compliance is affecting IT auditors' seeking and giving assistance via the listserv.
Several issues must be considered in this analysis. First, because the dependent variable is a
count, it is unlikely to follow a normal distribution so that Poisson or negative binomial regression
is recommended (Cameron and Trivedi, 1998). Poisson regression requires that the sample mean
and variance be equal; which assumption our data violate—a condition known as over dispersion.
We therefore conduct a negative binomial regression.
As can be seen in Table 8, the framework score is highly significant whether the dependent
measure is the number of threads in Panel A ( p = 0.0023) or the total number of posted messages in
Panel B ( p = 0.0043). Time period is not significant in either the analysis using the number of threads
18
Another way to consider analyzing the relation between the CobiT conceptual model and the message postings is to
look at characteristics of the messages such as quality, clarity, completeness and accuracy of the postings. These
measures, however, lack clear predictions. For example, an important but complex topic may result in inaccurate or
unclear postings because of its complexity. Conversely, the meaning behind a single thread is clear: i.e., a single issue or
idea that has evoked sufficient motivation to cause an auditor to seek or give help. For this reason, we only analyze
counts. Future research might explore the relation between audit risk and characteristics of message quality.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 257

Table 8
Negative binomial regression of thread and message counts on framework scores from Table 2
Information criteria df Estimate Chi-square p-value
Main effects
Panel A: number of threads per CobiT process a
Intercept 1 0.3383 0.35 0.5516
Framework score 1 0.4640 9.30 0.0023
Time period 1 0.0130 0.00 0.9817
Framework a time period 1 − 0.0330 0.05 0.8284

Panel B: number of messages per CobiT process b

Intercept 1 0.9780 2.52 0.1124
Framework score 1 0.4754 8.14 0.0043
Time period 1 0.4929 0.64 0.4237
Framework a time period 1 − 0.1303 0.61 0.4342
Note: A thread represents a stream of messages that relate to a particular topic.
a
Goodness-of-fit criteria: deviance Chi-square value of 67.0964 divided by 62 df = 1.0822.
b
Goodness-of-fit criteria: deviance Chi-square value of 68.2458 divided by 62 df = 1.1007.

or the number of messages. Recall that the framework score aggregates the expert panels'
assessments for complexity, client attention, client importance, and process risk according to each
information criterion and IT resource based on the indicators from Fig. 1. Hence, we view the
correlation between the framework score and practicing auditors' listserv behavior to provide
remarkable evidence consistent with Proposition 2 and consistent with our assertion that CobiT is a
sound framework when used in an IT audit setting.
Essentially, Proposition 2 proposes a link between CobiT and listserv behavior. Additional
evidence in support of such a link is obtained by regressing message counts (using negative
binomial regression) on the risk assessments associated with each CobiT process as supplied by
the expert panel and as shown in Table 2. As can be seen in Table 9, the risk assessments from the
expert panel predict the number of messages posted by the auditors ( p b 0.0001). The significant
relation between the risk assessment and the number of messages is not affected by the time
period in which the posting was made.

4.2. Proposition 3—CobiT importance predicts listserv behavior

Proposition 3 suggests that if the CobiT importance levels as shown in Fig. 1 are meaningful in
an audit context, then one expects that more important processes should result in more postings to

Table 9
Negative binomial regression number of listserv messages regressed on expert panel risk assessments
df Estimate Chi-square p-value
Intercept 1 − 8.1294 15.36 b0.0001
Time period 1 0.9756 0.22 0.6379
Risk assessments 1 1.4630 24.02 b0.0001
Time period a risk assessment 1 − 0.1334 0.20 0.6547
Note: Unit of Analysis = CobiT Process by Time Period. AI5, ME1, ME2, and ME4 added to CobiT after expert panel data
collected and therefore excluded from this analysis.
a
Goodness-of-fit criteria: deviance Chi-square value of 60.2062 divided by 56 df = 1.0751.
258 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 10
Negative binomial regression number of listserv postings regressed on CobiT importance rating
df Estimate Chi-square p-value
Panel A: message counts
Intercept 1 − 0.0155 0.00 0.9833
CobiT importance 1 1.2861 12.62 0.0004
Time period 1 0.1228 0.03 0.8683
Time period a importance 1 − 0.0854 0.06 0.8136

Panel B: thread counts

Intercept 1 − 1.7164 1.09 0.2973
CobiT importance 1 1.2867 15.00 b0.0001
Time period 1 − 0.2922 0.18 0.6707
Time period a importance 1 0.0539 0.03 0.8710

Panel C: least square mean number of postings per CobiT process by CobiT importance rating
CobiT importance rating Message count Thread count
Low 1.29 1.42
Medium 9.33 9.08
High 17.17 17.50
a
Goodness-of-fit criteria: deviance Chi-square value of 68.0897 divided by 62 df = 1.0982. CobiT importance ratings
are provided for each CobiT process within the CobiT framework as shown in Fig. 1.

the listserv. To test this proposition we regress the number of listserv postings (message and
thread counts) on the importance level associated with each CobiT process. Importance is coded
as a discrete variable with 1 = low, 2 = medium, and 3 = high.19 The time period in which the data
were collected is included as a control variable. As can be seen from Panels A and B of Table 10,
CobiT importance is highly significant for message counts ( p = 0.0004) and for thread counts
(p b 0.0001) and is unaffected by time period. This provides strong evidence to support
Proposition 3.

5. Discussion of the findings

Prior to discussing the findings, it is important to discuss some of the limitations and
strengths of the study. The study employed two separate sources of data to examine CobiT as a
theoretical framework pertinent to auditing (including operational, compliance, and financial
auditing). Each of these sources has limitations that are well recognized. For instance, the
survey data is limited because the participants are in a hypothetical context and might respond
differently than they would in practice. Additionally, the archival data is limited to only the data
that are available. However, an important strength of the study is that by using a combination of
data sources we mitigate the inherent limitations of survey and archival research methods when
used in isolation. Using the survey data we obtain various audit related assessments from a
panel of highly experienced IT auditors yielding consistent results. Using the archival data, we
obtain evidence from the ISACA listserv that is unobtrusive and reflects actual auditor behavior
in the field. By combining these two sources of data, we observe some remarkable correlations
when superimposed onto CobiT's conceptual model. When considered together rather than

19
Analysis with importance coded as a categorical variable produces similar results but its presentation is more difficult.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 259

separately, the two sources of data employed in this study produce consistent and compelling
evidence that CobiT's conceptual model is internally consistent and useful when applied to
auditing IT controls.
From a practical standpoint, the results of this study suggest that it is very important and
potentially very useful for the audit profession to seek academic examination of its practices.
Such examination provides highly needed evidence to policy-makers that either supports or
calls into question the conceptual foundations of current audit practice. In this case, the
findings suggest that the CobiT framework is significantly related to overall risk assessments
of the CobiT processes for which they are associated. Furthermore, the results indicate that the
CobiT framework can be used to predict the auditors' behavior in terms of seeking and giving
IT audit related help as reflected by the listserv postings. Together, these results should give
auditors and policy-makers assurance that CobiT is an appropriate supplement to COSO and in
an IT setting.
At the same time that our results generally support CobiT, the results also uncover some areas
that warrant additional scrutiny. In particular, seven of the thirty-four CobiT processes, as shown in
Table 5, have no associated listserv posting. This requires further study into how these seven
particular IT processes relate to audit settings. Furthermore, we find that the seven information
criteria and the five IT resources do not load cleanly on their respective categories. Rather, the IT
auditors in our study appear to think about these 12 items along three distinct dimensions. This
inconsistency is hardly surprising in that CobiT's original audience was primarily management
whereas its application today has changed towards its use as an audit framework for IT control.
This change in focus very likely changes the way that auditors think about information criteria and
IT resources.
We note that, at the present time, the accounting and information systems domains lack an
empirically validated theory of internal control in the sense of identifying the variables that
determine good control. Recognizing that the factor analysis reported in the paper is exploratory
and based on a relatively small sample, we conjecture on what the three factors may mean as a
start in building a preliminary theory of IT control. The first and strongest factor consists of four of
the seven information criteria and no IT resources. We interpret this factor to be an information
quality dimension as reflected by information reliability, confidentiality, and integrity obtained in
an efficient manner. We interpret the second factor to consist of IT processing considerations that
are strongly related to controls. That is, effective compliance with laws, regulations, and contracts
that is affected by people and by having the necessary data. The third factor we interpret as audit
considerations relating to IT design (i.e., applications and infrastructure), thus ensuring the
availability of information to the business. Extending this interpretation to a preliminary model of
internal control in an IT environment suggests that processes that affect information quality,
information processing, and system design directly impact the effectiveness of internal controls.
These dimensions are not too dissimilar from the CobiT framework, which suggests that business
requirements for information embody (1) quality requirements, (2) fiduciary requirements, and
(3) security requirements (IT Governance Institute, 2005).
A third aspect of our study, which suggests that CobiT needs additional work in an audit
setting, is that the analysis reported in Table 4 shows only a modest R-square (0.2138) when
relating the CobiT framework to risk assessments for IT processes. The low proportion of
explained variance may be the result of large individual differences in opinions about risk.
Alternatively, some significant variables that impact audit risk for IT processes may be
missing from the CobiT framework. One possibility may be that CobiT does not adequately
consider the environment outside the organization and how variables associated with the
260 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

competitive, legal, and economic environment might interact with IT processes. Along these
lines, CobiT has been criticized for being weak on security issues and companies typically
augment both the COSO and CobiT frameworks with specific security frameworks such as
ISO 17799.
We collected the data from the listserv over two different periods of time. The first period
covered the time immediately after Congress enacted SOX, whereas the second period covered
a period of years after the PCAOB released AS2. Although both periods are post SOX, it is
possible that auditors' response to internal control is still evolving. For this reason, we include a
variable in each analysis to control for the possible effects that the evolving audit environment
might have on the auditors' behavior. Throughout, we find no effect for the time period in which
the data were collected. One possible explanation for the consistency of behavior over this time
frame is that IT auditors were already highly engaged in IT control issues prior to or at the outset
of the post SOX era. IT auditors have been applying principles of internal control consistent
with the ideas of CobiT as it relates to operational, compliance, and financial auditing, because
internal controls are critical to the processing of information within and between organizations.
The fact that we find strong support for our propositions suggests that IT auditors were already
up to speed in terms of IT control issues (or at least became up to speed in a short amount of
time). Some might argue, however, that financial statement auditors without IT experience are
now wrestling with IT control issues. Hence, further research is needed to extend this study to
the time period after auditors have gained greater experience performing Section 404 audits and
to different groups of auditors with a specific focus exclusively on financial statement audits (or
the audit of internal controls as it relates to the financial statement audit) as opposed to auditing
in general.
In personal conversations between the authors and IT auditors about Section 404 work, the
auditors tell us that IT departments typically resist outside frameworks although auditing
standards clearly impose a requirement that the framework be publicly available. Efficiencies
should result by having IT departments and auditors sharing the same framework. Any framework
that is acceptable from not only a financial audit perspective but also from an operational
perspective is preferred to one that is only useful in the financial audit. COSO appears to serve
this need at a relatively general level. At an operational level, we note that CobiT was initially
developed as an IT benchmark consisting of best practices. From that context, we investigate the
appropriateness of CobiT to an audit setting. To the extent that CobiT is applicable as a dual use
framework; organizations can achieve efficiencies in either operations and/or IT audits through
its use. Issues related to efficiency and multiple use provide distinct possibilities for future
research.
Testing the internal consistency of the CobiT processes, information criteria, and IT resources
as they apply to an audit setting may be an important first step in developing a theory of internal
control. The CobiT framework conceptualizes and describes current practice with the aim of
helping us understand the domain. As defined by Merriam-Webster, a framework is “a basic
conceptual structure (as of ideas)” and as such closely resembles a theory. CobiT is a taxonomy of
IT processes and related controls that are asserted to affect certain information criteria and IT
resources. If the boundaries between processes are not valid in an audit setting, or if these processes
do not affect the information criteria or the IT resources that they are purported to affect, we would
have observed no significant results. Conversely, we tested and found support for these assertions.
By subjecting CobiT to the analysis in this paper, the present study demonstrates that the relations
and constructs within CobiT are relevant to an audit in a manner that one would expect a theoretical
model to behave.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 261

Considering our test of CobiT as a first step to developing a theory of IT control opens the
possibility for future research in several areas. These include studies into the types of control
deficiencies and their characteristics that give rise to variations in audit programs and audit
opinions. CobiT provides a means of classifying such control deficiencies and the results of this
study demonstrate that these classifications relate to various aspects of audit risk. Furthermore, we
only tested the core, basic structure of CobiT's conceptual model. CobiT contains other testable
constructs including a comprehensive and well-articulated maturity model for IT control. The
maturity model enables management of a company to evaluate and determine where on the internal
control quality spectrum their controls are currently located. We hope that the positive results of
this study will encourage others to begin a collaborative effort with the aim of developing a
comprehensive, validated, practical, and generally accepted theory of internal control as it relates
to information technology.

Acknowledgements

The authors contributed equally to the study and are listed in alphabetical order. This paper
benefited from helpful comments from Wendy Bailey, Mark Cecchini, Michael Cipriano, Uday
Murthy, Yi-Jing Wu and workshop participants at Brigham Young University, especially Scott
Summers, Mark Zimbelman and Doug Prawitt. We also thank Kelvin Liu and Yi-Jing Wu for
assistance with the data.

Appendix A. Concepts evaluated by expert panel

The following definitions were provided to the expert panel to ensure common understanding
of the concepts against which a “typical organization” was to be judged.
Complexity refers to the inherent complexity of the CobiT Information Criteria (IT Resources)
considered separately from the audit and in relation to a typical organization.
Client Importance refers to how critical each of the CobiT Information Criteria (IT Resources)
are to the mission of a typical organization.
Attention refers to the amount of time and resources the client devotes to each of the CobiT
Information Criteria (IT Resources) in a typical organization.
Process Risk refers to the typical organization associated with not achieving each of the CobiT
Information Criteria (IT Resources).

Appendix B. Information criteria and IT resources definitions from CobiT 4.0

Information criteria

To satisfy business objectives, information needs to conform to certain control criteria, which
CobiT refers to as business requirements for information. Based on the broader quality, fiduciary
and security requirements, seven distinct (and certainly overlapping) information criteria are
defined as follows:

• Effectiveness deals with information being relevant and pertinent to the business process as
well as being delivered in a timely, correct, consistent and usable manner.
• Efficiency concerns the provision of information through the optimal (most productive and
economical) use of resources.
262 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

• Confidentiality concerns the protection of sensitive information from unauthorized disclosure.

• Integrity relates to the accuracy and completeness of information as well as to its validity in
accordance with business values and expectations.
• Availability relates to information being available when required by the business process now
and in the future. It also concerns the safeguarding of necessary resources and associated
capabilities.
• Compliance deals with complying with those laws, regulations and contractual arrangements
to which the business process is a subject, i.e., externally imposed business criteria, as well as
internal policies.
• Reliability relates to the provision of appropriate information for management to operate the
entity and exercise its fiduciary and governance responsibilities.

IT resources

The IT resources identified in CobiT can be defined as follows:

• Applications are the automated user systems and manual procedures that process the information.
• Information is the data in all their forms input, processed and output by the information
systems, in whatever form is used by the business.
• Infrastructure is the technology and facilities (hardware, operating systems, database
management systems, networking, multimedia, etc., and the environment that houses and
supports them) that enable the processing of the applications.
• People are the personnel required to plan, organize, acquire, implement, deliver, support,
monitor and evaluate the information systems and services. They may be internal, outsourced
or contracted as required.

References

Cameron AC, Trivedi PK. Regression analysis of count data, econometric society monograph no. 30. Cambridge
University Press; 1998.
Colbert JL, Bowen PL. A comparison of Internal Controls: COBIT, SAC, COSO, and SAS 55/78. IS Audit Control J
1996;4:26–35.
COSO. Internal control—an integrated framework. The Committee of Sponsoring Organizations of the Treadway
Commission; 1992.
COSO. Enterprise risk management—integrated framework. The Committee of Sponsoring Organizations of the
Treadway Commission; 2004. http://www.coso.org.
Fedorowicz J, Gelinas Jr UJ. Adoption and usage patterns of CobiT: results from a survey of CobiT purchasers. IS Audit
Control J 1998;VI:45–51.
Hoffman T. IT auditors seek Sarb-Ox guidance. Computerworld 2004;38(15) April 12.
IT Governance Institute. Governance, control and audit for information technology. CobiT 3rd edition. Rolling Meadows,
IL: IT Governance Institute; 2000.
IT Governance Institute. “Control objectives, management guidelines, maturity models” in CobiT 4.0. Rolling Meadows,
IL: IT Governance Institute; 2005.
KPMG. S-O rules finalized. Frontiers in finance for decision makers in financial services; 2003. 10–15. November.
Lainhart IV JW. An IT assurance framework for the future. Ohio CPA J 2001 19–23. January–March.
Melville N, Kraemer K, Gurbaxani V. Review: information technology and organizational performance: an integrative
model of IT business value. MIS Quarterly 2004;28:283–322 June.
Merriam-Webster Online Dictionary; 2005. http://www.m-w.com/cgi-bin/dictionary?book=Dictionary&va=framework.
Messier Jr WF, Glover SM, Prawitt DF. Auditing & assurance services: a systematic approach. 5th edition. New York, NY:
McGraw-Hill Irwin; 2008.
 B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263 263

Netegrity. Sarbanes–Oxley. Regulatory compliance handbook; 2004. http://www.netegrity.com/PDFS/REGULATORY/

SOA%20Handbook%20Sheet.PDF.
Public Company Accounting Oversight Board. An audit of internal control over financial reporting performed in
conjunction with an audit of financial statements; 2004. http://www.pcaobus.org/documents/rules_of_the_board/
Standards%20-%20AS2.pdf.
Ramos M. Evaluate the control environment. J Account 2004;197:75–8 May.
Ridley G, Young J, Carroll P. COBIT and its utilization: a framework from the literature. Proceedings of the 37th Hawaii
International Conference on System Sciences; 2004.
Summerfield B. EU selects CobiT as an auditing standard; 2005. http://www.certmag.com.