Feature

Auditing Agile—A Brave New World

Chong Ee, CISA, CGEIT, is
a senior finance systems
manager with Twilio, a cloud
communications company
based in San Francisco, “We are running on Agile, so there is nothing
California, USA. Ee is focused to audit” is a refrain auditors hear all too often
on optimizing the use and when attempting to audit clients who use
integration of financial cloud Agile. For a profession rooted in plan-driven
www.isaca.org/currentissue
applications. Most recently, methodologies, from validating software
he implemented NetSuite development to documenting audit work papers, and this acknowledgment would have been ideal
and other Software as a Agile presents a unique conundrum.1 for proponents of Agile and Scrum, for whom
Service solutions at Trulia each two-week sprint would culminate in the
to support the company’s THE CASE AGAINST DOCUMENTATION demonstration of working software. Beginning
growth from startup through Conceived by 17 self-professed “organizational with a bare bones skeleton and inheriting more
initial public offering and then anarchists” in a Utah ski resort in 2001, the first features with each successive sprint, the Scrum
as a public company. Before two values of the Agile manifesto listed in figure team seeks to “burn down” the requirements
this, Ee spent 13 years in 1 appear to clash with common audit constructs, surfaced through the continual grooming of the
various compliance, audit as internal control design and validation are product backlog.
and consultant capacities for invariably predicated on process and procedural The waterfall model advocates for significant
Big Four audit firms, Fortune documentation. Furthermore, Scrum, a documentation throughout the development
500 companies and startups. popular iterative Agile software development life cycle. Some documentation does help to
Ee is a certified NetSuite methodology, advocates for self-organizing, cross- avoid any miscommunication on what has been
ERP Consultant and NetSuite functional teams, making audit challenging for agreed upon. Yet, it is the very maintenance
Administrator. auditors who are used to prescribed roles and of significant documentation during the
responsibilities that have clearly demarcated requirements phase that would, in turn, give rise
segregation of duties (SoD) to mitigate the risk of to more documentation, in the form of change
wrongdoing or fraud. requests seeking authorization for variances to
plan. Fundamentally, the Agile manifesto does
Figure 1—Manifesto for Agile Software Development not so much devalue documentation; rather, it
values working software more. Agile focuses
We are uncovering better ways of developing software
by doing it and helping others do it. Through this work on having good enough documentation to
Do you have we have come to value: initiate and sustain an open dialog among cross-
something • Individuals and interactions over processes and functional team members. The premise behind
to say about tools
• Working software over comprehensive having good enough, rather than comprehensive,
this article? documentation documentation is that, at the start of a project,
Visit the Journal • Customer collaboration over contract negotiation all that needs to be known is not yet known. A
pages of the ISACA • Responding to change over following a plan
plethora of unanticipated outcomes can arise; for
web site (www.isaca. That is, while there is value in the items on the right, instance, customers can, and often do, change
org/journal), find the we value the items on the left more. their minds on features, even as the software is
article and choose Source: agilemanifesto.org. Reprinted with permission. being coded (64 percent of features developed
the Comments tab to
never or rarely get used).3 Therefore, having
share your thoughts.
To understand the evolution of Agile and excessive documentation at the start and using
Go directly to the article: Scrum and identify related implications for it as a benchmark for downstream activities can
audit, it helps to go back to the inception of the seem counterintuitive.
waterfall model, first proposed in 1970. Even Despite a lesser amount of documentation,
though the waterfall model defines distinct Agile can actually create greater transparency on
phases for managing the development of large uncertainties that may not be otherwise visible
software systems, it nonetheless acknowledges during a project’s infancy. According to Jens
the need for iteration.2 Fast forward 30 years Østergaard, founder of House of Scrum, the

1 ISACA JOURNAL VOLUME 2, 2016

risk associated with identified uncertainties tapers off with to be met for a story to be delivered. Acceptance criteria
each successive sprint, whereas with waterfall, the initial, identification can also break up bigger stories into smaller,
comprehensively documented set of specifications may give more digestible pieces for developers to consider. User stories
a false sense of security, only to be undermined when hidden typically follow the independent, negotiable, valuable to
complexities surface downstream when the project enters users or customers, estimable, small and testable (INVEST)
the testing and go-live phases (figure 2).4 From an audit set of attributes.
perspective, an auditor who looks for evidence too soon is Behavior-driven development (BDD) is a means for
likely to be reviewing material that is later overwritten as discovering and, consequently, testing against what the
requirements become further clarified. software ought to do. “A story’s behaviour is simply its
acceptance criteria—if the system fulfills all of the acceptance
Figure 2—Risk Transparency in Waterfall vs. Agile criteria, it is behaving correctly.”5 Automated testing tools
allow the team to describe the acceptance criteria in terms of
Waterfall
Risk scenarios; scenarios become automated tests with the addition
of step definitions using code. Scenarios take the form
illustrated in figure 4.
Agile
As Project Progresses
Figure 4—Scenario Template
Source: Chong Ee. Reprinted with permission. Given <some initial context>
When <an event occurs>
FROM STORYTELLING TO STORY-TESTING
Then <ensure some outcomes>
This is not to discourage auditors from intervening early in the
Source: Chong Ee. Reprinted with permission.
development of software. In fact, an unmined opportunity lies
in user stories that are developed to characterize specifications
To illustrate, consider the example of developing a
in Agile. Figure 3 depicts a template for user stories.
purchase requisition application for expenses over
US $1,000 (figure 5).
Figure 3—User Story Template
User Story Template
Figure 5—User Story for Purchase Requisition
As a <role>
Purchase Requisition Submission
I want to <action>
As a buyer
So that <benefit>
I want to submit a purchase requisition for approval
Source: Chong Ee. Reprinted with permission.
So that I can contract with the vendor for services.
Source: Chong Ee. Reprinted with permission.
One of the key ways Agile encourages responding to a
change rather than following a plan (see the fourth value
How can it be confirmed that the Scrum team has
listed in figure 1) lies in the active update of remaining
delivered on this user story? A scenario to consider is shown
user requirements; in Agile, this is known as the grooming
in figure 6.
of the product backlog of user stories for each sprint.
Scenarios force stakeholders to clarify just exactly what
The user stories represent the voice of the customer, so
they need and can help to mitigate the risk of gold plating,
the development team is wise to ensure that they are
which is the addition of features that do not add value.
accommodated to the greatest degree possible.
Thus, auditors can get involved early in the software
How does the Scrum team know they have delivered on
development process not by looking for comprehensive
a user story? Behind every user story is a set of acceptance
documentation upfront, but rather by taking part in the user
criteria that helps clarify the specific conditions that need
story development.
ISACA JOURNAL VOLUME 2, 2016 2
 Figure 6—Positive Scenario in Submitting Purchase Requisition Within each story, there is an opportunity to craft an
Scenario 1: Requisite Information Exists abuse scenario where the nature of the proposed validation is
Given that The fields “department,” “general ledger account,” negative, i.e., covering scenarios that do not follow the plan.
“start and end dates” and “amount” are completed Figure 7 describes the expected application behavior for
And The vendor selected is from the authorized list five abuse scenarios. Figure 8 maps these scenarios against
And The purchase approver defaults to the user’s supervisor traditional internal control objectives.
And The purchase approver is an active user
And The purchase approver has approval authority Figure 8—Mapping Scenarios to Control Objectives
And The amount is greater than US $1,000 Scenario Completeness Accuracy Validity
When A user submits a purchase requisition
2 X
Then The purchase requisition is routed to the purchase
approver 3 X
And An email is sent to the purchase approver to log into 4 X
requisition application.
5 X
Source: Chong Ee. Reprinted with permission.
6 X
Source: Chong Ee. Reprinted with permission.
Figure 7—Negative Scenarios in Submitting Purchase Requisitions
Scenario 2: Missing Information
It turns out that there are no controls addressing whether
Given that The fields “department,” “general ledger account,”
“start and end dates” and “amount” are missing or not the proposed spending request is recorded correctly
When A user submits a purchase requisition to the right general ledger (GL) account. Therein lies an
Then An error message pops up to remind the user of opportunity for auditors to articulate other roles needed to
mandatory fields to complete. ensure transaction integrity. By participating in the Scrum
Scenario 3: Amount Under US $1,000 team during, rather than after, the sprint, auditors can add
Given that The amount is under US $1,000 a story for finance personnel to provide a second layer of
When A user submits a purchase requisition review, as outlined in figure 9.
Then An error message pops up to remind the user of the
purchase requisition policy.
Figure 9—Finance Review of Purchase Requisitions
Scenario 4: No Approval Authority
Purchase Requisition Finance Review
Given that The purchase approver does not have approval authority
As a Finance user
When A user submits a purchase requisition
I want to Review an approved purchase requisition
Then An error message pops up to contact the application
So that I can ensure that it has been recorded to the correct
administrator.
general ledger account.
Scenario 5: No Independent Approver
Scenario 7: Correct Selection of GL Account
Given that The purchase approver is the same as the user
Given that The general ledger account has been correctly coded
When The user submits a purchase requisition
When The purchase approver approves the purchase requisition
Then The user is prevented from submitting the requisition.
Then A finance user checks the “finance approved” checkbox.
Scenario 6: Duplicate Requisitions
Scenario 8: Incorrect Selection of GL Account
Given that The amount and vendor selected are the same as an
Given that The general ledger account has been incorrectly coded
existing requisition submitted on the same date
When The purchase approver approves a purchase requisition
When A user submits a purchase requisition
Then A finance user edits the general ledger account
Then An error message pops up to warn the user of a
possible duplicated requisition. And An email is sent to the requisitioner describing the
Source: Chong Ee. Reprinted with permission. account update
And The finance user checks the “finance approved” checkbox.
Source: Chong Ee. Reprinted with permission.

3 ISACA JOURNAL VOLUME 2, 2016

 Had the development approach been waterfall, auditors
would have been satisfied obtaining sign offs of the initial
requirement to have the enterprise buyers correctly code their
proposed purchase to the right general ledger accounts, even • Learn more about, discuss and collaborate on audit
though, in reality, this process is not executable. In addition to tools and techniques in the Knowledge Center.
participating in user story development, auditors need to also www.isaca.org/
adapt the manner in which they perform audits.
A specification should do something.6 Thus, when auditing
topic-audit-tools-and-techniques
Agile, instead of expecting a three-ring binder of written
specifications, a more appropriate approach may be for ROLES THAT OVERLAP
auditors to request the system log of executable specifications. The remaining Agile manifesto value addressed in this article
The following questions can help the auditor gain insight emphasizes customer collaboration over contract negotiation
on specifications: (see the third value in figure 1). With its emphasis on distinct
• Where are the automated test results? phases and handoffs, the waterfall model can be likened to
• What happens when they fail? a relay race,9 while Scrum is derived from a restarting play
• What is the test coverage? where players huddle with their heads down to attempt
• How often are automated tests updated? to gain possession of the ball. When applied to software
The good news is that Agile and Scrum artifacts are development, the Scrum approach focuses on having the
not vastly different from the traditional audit artifacts that distinct phases overlap through collaboration..
auditors rely on to evidence a system of internal controls. User What implications does this have for audit? A key precept
stories mirror user narratives, and acceptance criteria mirror behind the emergence of design thinking as a means to
application controls that ensure the accuracy, completeness solving problems is the emphasis on collaboration to attain
and validity of the transactions processed. Auditors who sustainable product design. By playing a key role in the Scrum
consider a more complete picture of varied roles and scenarios team by considering abuse scenarios in user stories, auditors
(illustrated through user stories and acceptance criteria) make engage actively in the collaboration.
a valuable contribution to the Scrum team. Another useful adjustment auditors can make is to expand
Auditors can also add value by challenging the prevailing their scope of auditees. As mentioned earlier, the product owner,
mind-set that tests at the end of a project are the only way to insofar as he/she drives the product backlog of user stories, is a
produce the required quality. It is widely accepted that the key resource in making sure that compliance and security needs
later a bug is detected, the more costly it is to fix.7 In the same are met. Auditors can make a valuable contribution by ensuring
manner, if acceptance criteria of stories are developed as a broad representation of stakeholders on the Scrum team. The
part of BDD, developers can employ test-driven development composition of the Scrum team plays a key role in aligning
(TDD) to write tests before writing code. Studies on the expectations for the project among all those involved.
adoption of TDD in three Agile teams at Microsoft and one However, just because the Scrum team is cross-functional
at IBM reveal a 40 and 90 percent decrease, respectively, does not mean all development, test/staging and production
in the prerelease defect density of products.8 Further, when environments are accessible to anyone on the team. The audit
combined with a continuous integration (CI) server that objective behind SoD—restricting access by environment
triggers testing every time new code changes are checked or configuration so that no one single individual has the
in, testing becomes part of, or a constant accompaniment ability to circumvent or hack the prevailing system of internal
to, coding, as opposed to a separate downstream test controls—is not mutually exclusive from the development
phase. Consequently, the earlier identification of unit and objective of maintaining code integrity, which is making sure
functionality errors reduces the cost to fix them later and frees that valid code changes are not inadvertently overwritten or
up time for testers to add value through exploratory and end- diluted by competing ones that have not been tested or do not
to-end transaction testing. integrate well with existing interfaces.

ISACA JOURNAL VOLUME 2, 2016 4

 With the emphasis on customer collaboration over CONTROL READINESS IS A FUNCTION
contract negotiation, it can be easy to be misled into How control-ready an enterprise is depends not just on the
thinking that Agile favors process over outcome. Interviews practices adopted (whether that is waterfall or Agile in software
conducted with teams from three organizations, one using development), but also on the environment, which is unique
waterfall, another using Agile and a third using a hybrid of to each enterprise. Whether management employs a more
both, revealed that the waterfall organization saw a greater top-down or bottom-up approach in controlling software
emphasis on product or outcome with preventive controls, development can make a difference in whether waterfall, Agile
whereas the Agile organization leaned toward process with or a combination of both is ultimately embraced. In effect,
detective and corrective controls.10 With Agile’s fundamental auditors can audit the Agile team on how well it does Agile.
focus on outcome—working software—one can make a For far too long, auditors have been relying on validating
counterargument that in Scrum, control practices such as the operating effectiveness of processes in the hope that a
TDD, CI, and automated unit and acceptance testing are carefully controlled process will yield a positive outcome,
really focused on outcome, whereas the waterfall model, e.g., verifying evidence of sign-offs. The problem with process
with its focus on formalized signoffs, is really borne out of an controls is that while they may be necessary for fostering a
emphasis on process. Figure 10 illustrates the different types positive outcome, they are by no means sufficient. Consider
of process- and outcome-based controls in Agile and Scrum. the user story of submitting purchase requisitions for approval
outlined in figure 9.
Figure 10—Agile Outcome vs. Process Controls When auditors sample invoices, they may find that all have
been matched with approved requisitions. They are able to find
Outcome Controls Process Controls
a 100 percent two-way match with no exceptions. It would
Percent of test coverage Product backlog
appear that all purchases made have been approved beforehand.
Number of failed builds Progress in burndown chart Yet they cannot help but notice from the system audit
Number of failed unit tests Findings from sprint timestamps that for a particular group of buyers, requisitions
retrospectives
are almost always created shortly before invoices are applied—
Number of failed acceptance tests Composition of Scrum team sometimes a matter of mere minutes. It turns out, for this group
Source: Chong Ee. Reprinted with permission. of buyers, because proposed spend is highly volatile and hard
to predict upfront, requisitions are created only upon receipt of
As for whether Agile has more detective or corrective invoices, rather than before receipt of service.
controls than waterfall, there are shades of gray when it comes These illustrative audit findings cast doubt upon requisition
to labeling a control as preventive or detective. Because the approval as a means to control spend precisely because it is
project is seen from production, the testing phase from the not so much performed before services are rendered as it is
waterfall model is preventive, i.e., it reduces the likelihood of after receipt of services—and often as a means of generating
coding errors from arising in production by uncovering them evidence for audit. Auditors must be proactive to ensure that
in a test environment. From the perspective of development, audits remain effective safeguards against errors or fraud, not
however, testing is detective as it uncovers errors in coding from ritualized practices of audit for audit’s sake.11 Agile, with its
the development phase. Likewise, TDD in Scrum and Extreme emphasis on working software, focuses on outcome and, thus,
Programming (XP), by forcing developers to fail the test before provides auditors the opportunity to adapt their approach
the code is written, detects the error even as it prevents it accordingly. A widely circulated observation among Agile and
from arising ultimately in production. The same argument can Scrum circles is that the Standish Group’s study of software
extend to automated unit and acceptance testing; while they projects conducted between 2002 and 2010 revealed that
detect errors in development and staging environments, they Agile is three times more likely to succeed than waterfall.12
really prevent them from arising in production. To characterize Yet, auditors point out that the same Standish Group study
Agile controls as more detective than preventive is to miss an reported that both Agile and waterfall projects shared a 50/50
opportunity to dive deeper into what truly goes on. chance of being challenged.

5 ISACA JOURNAL VOLUME 2, 2016

 As lightweight frameworks, Agile and Scrum are not 3 Johnson, J.; Standish Group Study reported at XP2002
intended to be comprehensive. They do not address risk 4 Ostergaard, J.; “Why Scrum Is So Hard,” 20 August 2009,
management, product strategy and other areas that comprise http://www.slideshare.net/marakana99/jens-stergaard-on-
the slew of activities to enable and sustain product launch, why-scrum-is-so-hard-2416688
continual enhancement and maintenance. When auditing 5 North, D.; “Introducing BDD,” Better Software,
Agile, it is important for auditors to realize that enterprises March 2006, http://dannorth.net/introducing-bdd/
employing Scrum or Agile are not running on empty—there 6 Ibid.
are indeed artifacts and ceremonies from a process perspective 7 Beohm, B.; V. Basili; “Software Defect Reduction Top 10
and metrics to track test coverage and automated test results List,” Computer, vol. 34, iss. 1, January 2001,
from an outcome perspective. Perhaps, more important, p. 135-137
auditors are best served by seeing that Agile, like its waterfall 8 Nagappan, N.; E. Maximilien; T. Bhat; L. Williams;
predecessor, is not a silver bullet to resolving the age-old “Realizing Quality Improvement Through Test Driven
struggle of bridging compliance and security with software. Development: Results and Experiences of Four Industrial
Teams,” Empirical Software Engineering, vol. 13, iss. 3,
ENDNOTES June 2008
1 This article uses several Agile terms that are critical to 9 Takeuchi, H.; I. Nonaka; “The New Product Development
understanding. A sprint is a set period of time during Game,” Harvard Business Review, 64, no. 1,
which specific work has to be completed and made ready January-February 1986
for review. A burndown chart is a graphical representation Cram, W. A.; M. K. Brohman; “Controlling Information
10

of work left to do vs. time. A product backlog is a Systems Development: A New Typology for an Evolving
prioritized features list containing short descriptions of all Field,” Information Systems Journal, 23 (2), 2013,
functionality desired in the product. p. 137-154
2 Royce, W.; Managing the Development of Large Software 11 Power, M.; The Audit Society: Rituals of Verification,
Systems, Technical Papers of Western Electronic Show and Oxford University Press, UK, 1997
Convention (WesCon), 25–28 August 1970, Los Angeles, The CHAOS Manifesto, The Standish Group, 2012
12

California, USA

ISACA JOURNAL VOLUME 2, 2016 6