Why COSO is

Organizations of the Treadway Commission

(COSO) approach.1 The recently released
COSO framework sets the standards for
enterprise-wide risk management (ERM).
COSO views ERM as a process aimed at

flawed helping organisations identify potentially

adverse events and subsequently manage the
associated risks in furtherance of the entity’s
business objectives. When applied to
operational risk management this is often
COSO not only fails to help a firm assess its translated to mean: begin with a
comprehensive survey of the organisation to
risks, it actually obfuscates the risk identify, define and assess the full spectrum of
‘risks’ in each business’ underlying processes.
assessment process. By Ali Samad-Khan Then define a series of responses or controls
to mitigate the risks that threaten to prevent
the entity from meeting its objective. This is
often accomplished by establishing a list of

O
perational risk is one of the most issues and follow-up action plans to ensure
significant risks that businesses face compliance with this programme can be
in today’s complex global economy. verified over time through the audit process.
For most of the world’s leading institutions it At a macro level, this approach appears both
has become more than apparent that comprehensive and sound, but the devil is in the
implementing an effective operational risk details and the specious logic underlying COSO
management programme can help reduce becomes evident during implementation. While
losses, lower costs associated with fixing COSO may help organisations identify and
problems and increase customer and resolve some of their more obvious control
employee satisfaction, thereby improving weaknesses, in our view, it is completely
financial performance and enhancing inappropriate for use in operational risk
Ali Samad-Khan shareholder value. management. Fundamentally, COSO is
Basel II may have forced banks to review inappropriate for use in operational risk
their approach to managing operational risk, management because the definition of risk used
but for most leading institutions the question under this approach is wholly inconsistent with
was never whether to establish such a the definition of risk used in the risk
programme, it was how. But many institutions management industry and by the BIS (see next
are still unsure of the benefits. Some are still section for a full explanation of this point). In
struggling to decide whether to comply with addition, the method COSO prescribes for an
the BIS basic indicator, standardised or organisation to assess its risks is highly subjective,
advanced measurement approach. overly simplistic and conceptually flawed.
Nevertheless, compliance issues aside, most COSO not only fails to help a firm assess its
banks have come to the conclusion that if they risks, it actually obfuscates the risk assessment
are going to have to establish an operational process. Because risk assessment is a
risk management programme then they want it foundational element in the risk management
to be based on a sound framework. What is process, and because COSO yields an entirely
perhaps surprising though is that while we are counterfeit set of risks, the spurious and
many years into this process, there is still no misleading results of the flawed risk
industry consensus on what shape or form this assessment stage contaminate every
framework ought to take. And while there has subsequent stage of the process. As a result,
been much heated debate on this issue, much the recommended risk mitigation strategy –
of it has been based on personal opinion and the set of controls and action plans designed
not fact. This is because, even today, a number to mitigate the identified risks – is likely to be
of fundamental misconceptions exist about the non-optimal at best. In the worst case, it may
true meaning of operational risk management lead organisations to expand and intensify
in its modern conception. The purpose of this control structures in areas where they are
paper is to shed light on one of the main issues already over-controlled, while completely
that is driving this confusion. ignoring areas of major control weakness,
Many people believe that managing leaving the organisations both oblivious and
operational risk can be accomplished by vulnerable to huge operational losses that
following the Committee for Sponsoring could hit them like a bolt from the blue.

January 2005 www.operationalriskonline.com 1

 In our view, COSO – as it is currently historical losses, is much more likely to be
applied – is a wholly inappropriate approach aware of the full range of potential risks
for managing operational risk; it is a huge affecting a business (and their relative
waste of resources and is very likely to do probabilities) than is a business manager.
more harm than good. Having a qualified risk manager ask a business
One obvious issue with COSO is that it is manager where his or her major risks are is
hugely resource-intensive. This is because similar to having a doctor ask his or her
COSO requires that all processes be assessed, patient: to which major diseases do you think
irrespective of their individual contributions to you are most exposed? Some patients will know
the organisation’s total risk (because one the answer, but most will not, which is why
cannot know the level of contribution to total they went to see their doctor in the first place.
risk without first conducting a risk- In a well-managed organisation, the risk
assessment). Identifying and documenting the professional should serve as the doctor and the
risks in each and every process could take business manager as the patient.
many person-years. One mid-sized bank For those who still believe the right
recently estimated that it would require 192 approach is to ask business managers to self-
person-years to complete such an assessment assess the risks within their organisation’s
across the entire organisation. Clearly, the cost underlying processes, we ask what would they
of such massive resource commitment is not have recommended to the Governments of
something an organisation can easily absorb, India, Indonesia or Sri Lanka, prior to the
particularly if the exercise needs to be repeated recent tragedy, considering that tsunami risk
on an annual basis, which is necessary because probably was not a recognised risk in any of
for operational risk management to be their processes?
effective it must be implemented through a Another major problem with COSO is that
dynamic process with continuous monitoring. a typical risk-assessment implementation
Still, this is not a problem as long as the cost generally produces a huge catalogue of risks –
can be justified. often in the thousands. Thus, when it comes
A serious problem with COSO has to do to actually managing these risks across an
with the way the ‘risk’ information is organisation, ie, determining which risk
collected. The starting point under COSO in mitigation strategy is optimal, it is very
a typical implementation is the identification, difficult to prioritise actions because without
definition and assessment of risks in a a ‘normalised’ rank ordering of risks one
business process. In general, the persons cannot know which controls should be given
interviewed are business managers. While precedence in implementation.
these persons may be well qualified to run To address this problem, COSO developed
their own businesses, they do not necessarily the likelihood-impact method of risk
know anything about risk. Yes, they can assessment. Under this approach, businesses
probably come up with a long list of potential calculate the magnitude of their risks based on
risk scenarios, but that’s only half the story. a mathematical formula, where risk is equal to
To know which risks are real risks the the likelihood that a given event will occur
manager would also have to know the relative multiplied by its effect (impact), should it
probability of each ‘risk event’ that could occur, such that, Likelihood x Impact = Risk.
affect his or her business. After all, a tsunami
and a wire-transfer error are both risks, but
without knowing whether a 99% level tidal
wave or 99% level fat-finger error could do
FOR those who understand the
concept of risk, as it is used in the
risk management industry, it is clear that there
more damage – in the context of their is something fundamentally wrong with this
existing control environment – they cannot approach. Using the COSO formula the
know which risk poses a greater threat. And worst-case outcome is characterised by high
as it turns out there is often a major likelihood and high impact; however, under
discrepancy between perception and reality. the risk management approach, the worst-case
The only way one can identify one’s real risks outcome is characterised by a low probability
is by studying historical loss data. A risk (low frequency) – high impact (high severity)
manager, whose job it is to know about event, such as a $1 billion dollar unauthorised
trading loss. In fact, there is no such thing as
1
COSO’s objective was to help standardise procedures for enterprise risk a high likelihood (high frequency) – high
management by developing a conceptually sound framework providing impact (high severity) event. This would
integrated principles, common terminology and practical
implementation guidance. For more information on COSO, visit
characterise a risk (type of loss) that occurs
www.erm.coso.org hundreds of times a year and each time causes

2 January 2005
billion-dollar losses. This is clearly a phantom risk. What’s even
worse is that COSO also completely understates the one area 1. COSO vs reality: COSO produces false positives
of real risk. In summary, the COSO approach to risk and false negatives
assessment will tell you your risk is very high in areas where
you have no risk, and will also tell you that you have moderate Risk management industry Coso
risk in the very area your risk is of the highest order. Simply
stated, COSO produces both false positives and false negatives. High (3) n/a n/a High (3) 3 6 9 Phantom risks

The contrast is illustrated in figure 1 (right).

Likelihood
Likelihood
Med (2) n/a Med (2) 2 4 6
Some advocates of COSO have suggested that this problem
only exists when the analysis is qualitative or high level. They COSO COSO
Low (1) Low (1) 1 2 3 Real risks
argue that likelihood and impact analysis works well when the
Low (1) Med (2) High (3) Low (1) Med (2) High (3)
inputs are expressed in more quantitative terms, such as
Impact Impact
percent probability and dollar magnitude. To examine this
argument, let us express it in the context of a simple business
problem. Suppose you want to know the risk associated with
your having a car accident during the coming year. If you example, if we were to think of a ‘risky event’ as a 99% level
know that you have a 10% chance of having an accident and (1% likelihood) event, then from the table shown above one
you expect that accident will cost $10,000, then you would can see that this would correspond to a $50,000 loss. But
calculate your risk as follows: when one defines risk as the product of likelihood and impact
one can see the $50,000 x 1% ($500) event would imply less
Likelihood x impact = risk risk than the $25,000 x 5% ($1,250) event. This wholly absurd
Risk 1: 10% x $10,000 = $1,000 rank-ordering clearly demonstrates that, far from improving
operational risk management, COSO obfuscates the process of
But as you further consider this matter you realise that the determining an organisation’s true risk profile.
problem is more complex than originally perceived. After all, So what should we do now? Suppose now we were to take a
the 10% likelihood relates only to a $10,000 event; and there weighted average of all the risk-results drawn from the table
is also a possibility, let’s say a 1% probability, that you could above? What would that answer represent? The mean of all
have a very bad accident, which could result in the total the risk results would equate to probability weighted severity
destruction of your $50,000 car. Therefore you have two (which seems to equate to mean severity). But if this is true
possible ways of estimating your risk, as shown below: then we have a problem, because mean severity is somewhat
similar to expected loss (mean aggregate loss), whereas the
Likelihood x impact = risk risk management industry and BIS definition of operational
Risk 1: 10% x $10,000 = $1,000 risk equates to the unexpected loss. Without knowing
Risk 2: 1% x $50,000 = $500 anything else about COSO it is clear that the meaning of risk
under COSO is altogether inconsistent with the true meaning
What becomes immediately apparent is that two completely of operational risk. By following the COSO definition of risk,
valid assessments can yield different risk results. In fact, upon one is shooting at the wrong target, one that is not even a
further consideration, it becomes evident that the problem is close approximation!
still more complex because there are, in fact, multiple What is fundamentally wrong with the COSO-based risk
‘solutions’, because there are potentially an infinite number of assessment approach is that the question is flawed. Instead of
likelihood and impact combinations, as shown below: looking for the product of likelihood and impact we should
be taking as the results of this process the full set of likelihood
Likelihood x impact = risk and impact combinations. And if we were to plot them on a
Risk 1: 10% x $10,000 = $1,000 graph we would get something similar to what we see in
Risk 2: 1% x $50,000 = $500 figure 2 (right).
............ When you connect the dots, the full set of combinations
Risk 999: 5% x $25,000 = $1,250 would represent a set of points on a continuum. This is
Risk 1,000: 20% x $6,000 = $1,200 known in actuarial science as a severity distribution.
And what does one do with this severity distribution? Will the
From the outcome, one can clearly see that all the ‘risk- severity distribution give us the answer we are looking for? If we
results’ are banded together (from $500–$1,250) with little look at the 1% probability event on this distribution, will that
differentiation. This is because the higher the impact the lower not tell us our level of risk? No, not quite yet. As it turns out the
the likelihood (an incremental gain in likelihood offsets any severity distribution is just one piece of this puzzle. Returning to
corresponding reduction in impact). The major differences in our example, the severity distribution is a distribution of single-
the risk-results are due to the fact that the product of two event losses, showing the full set of losses and corresponding
figures near their respective means is greater than the product probabilities associated with a single car accident. But this is not
of two figures at opposite extremes. But this is an idiosyncrasy what we want. We want to know our operational risk in terms of
of the arithmetic process and is not reflective of any legitimate the total amount of money we could lose from all the car
difference in the level of risk; in fact, the opposite is true. For accidents we could have in the next year. For this we also need

© 2005 Incisive Media Investments Ltd. Unauthorised photocopying or facsimile distribution

of this copyrighted newsletter is prohibited. All rights reserved. ISSN 1741-8291
3
 Industry viewpoint COSO

to know how many accidents we could have in is already over-controlled, while completely
one year – or more precisely a probability ignoring areas of control weakness. As we all
distribution for the number of accidents we know, the consequences could be disastrous.
could have in a given year. This is known in
actuarial science as a frequency distribution. Operational risk management in its
Under the risk management industry and modern conception
BIS definition, operational risk – as shown An effective operational risk management
above – is described in the context of an programme requires a sound framework. The
aggregate or total loss distribution, which is goal of this framework should be to provide
a convolution (a mathematical combination) reliable information to key decision-makers so
of both a frequency and a severity that they are aware of their most significant risks
distribution, where the relevant points are
the expected loss and, more importantly, the

“ SOME MAY STILL ARGUE THAT COSO IS

unexpected loss. The expected loss is the
total amount of money one expects to lose
USEFUL BECAUSE IT IMPROVES BUSINESS
in a year, on average, and the unexpected
loss is the total amount of money one could
lose in a very bad year (at a specified
confidence level) in excess of the average.
PROCESS MANAGEMENT; EVEN IF THIS IS TRUE,
IT SHOULD NEVERTHELESS BE CLEAR
THAT BUSINESS PROCESS MANAGEMENT IS
“
(For a technical explanation of the terms NOT OPERATIONAL RISK MANAGEMENT
expected and unexpected loss, please refer to
the BIS guidelines).
From the above discussion it should be as well as the quality of their corresponding
evident that the risk result under the internal controls, information that will allow
likelihood-impact approach equates to mean them to make educated decisions when
severity, which is completely unrelated to the developing risk management, risk mitigation
term risk as it is defined by the risk and risk transfer strategies. Managing
management industry and the BIS. In fact, operational risk fundamentally revolves around
mean severity multiplied by mean frequency the process of optimising the risk-control
gives you the mean aggregate loss – the relationship in the context of cost-benefit
expected loss. Whereas the real measure of analysis. This, in turn, requires a process for
risk is the unexpected aggregate loss. accurately monitoring (measuring) each
The likelihood-impact analysis approach to business’ changing risk and control profile.
risk assessment can be summarised as follows: To accomplish this goal four things must be
it is based on a process whereby one asks the done correctly. First, the risk management
wrong people to answer the wrong question, department must be able to provide managers
which in any case is a flawed question, with objective information to help them
because it has an infinite number of different, better understand where their risks really are,
but theoretically valid answers. And even if not ask them to guess where their risks might
you were to ignore the answers and take only be. Fundamentally, one cannot manage one’s
the potentially useful information from this
process – the full set of input pairs – you
would still only have one part of the solution. 2. Severity distribution
No matter how you sum it up, four wrongs
don’t make a right.
In summary, implementing COSO requires
Probability
a gargantuan effort, and, in the context of
(20%, $6,000)
operational risk management, it produces
spurious and misleading results. Acting on
this information may divert managers’ (10%, $10,000)
attention from their real risks and instead (5%, $25,000)
focuses their attention on phantom risks,
while at the same time providing them with a (1%, $50,000)
false sense of security. Furthermore, any risk
mitigation strategy based on this flawed risk
0-10 10- 20- 30- 40-
information is likely to focus attention and
20 30 40 50
resources on the wrong controls. It is highly
conceivable that this approach could lead to Impact
an intensification of controls where a business

www.operationalriskonline.com 4
 operational risks without measuring one’s Fourth, one needs to institute a
operational risks. It is very difficult to be able comprehensive and fully transparent
to differentiate between major risks and minor monitoring and reporting process with built-
risks and real risks and phantom risks without in incentives to encourage desired
being able to accurately measure these risks in behavioural change.
the first place. It is also impossible to develop It is difficult to think of ways one could
an effective risk management programme even begin to manage operational risk
without knowing which risks must be dealt without having these foundational elements
with as a top priority. in place. Best-practice calls for an integrated
Second, one must help managers operational risk measurement-management
understand how well their real risks are being programme, whereby objective, transformed
managed through their existing set of (normalised) measures are used to identify
controls, so they can know where they are levels of risk and internal control quality. But
for these measures to be meaningful they
must be based on reliable information,

“
specifically: internal and external loss data,
MANAGING OPERATIONAL RISK
FUNDAMENTALLY REVOLVES AROUND THE
PROCESS OF OPTIMISING THE RISK-CONTROL
RELATIONSHIP IN THE CONTEXT
“ theoretically valid risk measurement and
assessment, objective control self-assessment,
validated risk indicators, appropriate follow-
up action results, disciplined scenario analysis
OF COST-BENEFIT ANALYSIS and well founded VAR calculation.
Can this really be done and is it practical?
The answer is yes to both questions, but only
if the underlying framework is based on sound
over-controlled and where they are under- reasoning, which must in turn be based on a
controlled in the context of their overall comprehensive understanding of the issues.
operational risk strategy and risk (loss) And these issues must be addressed logically
tolerance. One cannot have a zero-tolerance and objectively, one issue at a time.
policy towards operational risk, just as one
cannot institute perfect controls. An Conclusions
organisation has to be realistic in establishing COSO was initially conceived in the early
a level of risk and loss tolerance. 1990s, and for a long time represented best
Third, one needs to determine what level of practices in enterprise risk management. Then
controls is appropriate after having banks began collecting historical loss data,
conducted a circumspect analysis of the and we entered the dawn of a new age. As
associated costs and benefits of each risk the process of collecting loss data became
mitigation and transfer strategy. more widespread, thanks to the bold

3.The actuarial approach

individual risk matrix for loss var total loss

loss events loss data distributions calculation distribution

74,712,345 P
Frequency
74,603,709 of events
74,457,745
Employm BusinesExecution,
Clients, PrDamage to
Internal Fraud
External Fraud
Practice Physical Assets Disruptio & Proce
Business P Assets
Workplac System F Managem

Corporate Finance
Number 36 3 25 18 36 33 150 2

74,345,957
Mean 35,459 52,056 3,456 45,678 56,890 56,734 1,246 89,678
Standard Deviation 5,694 8,975 3,845 4,567 7,890 3,456 245 23,543
Trading & Sales
Number 50 4 35 25 50 46 210 3
Mean 53,189 78,084 5,184 68,517 85,335 85,101 1,869 134,517

74,344,576
Standard Deviation 8,541 13,463 5,768 6,851 11,835 5,184 368 35,315

VaR
Retail Banking
Number 45 4 32 23 45 42 189 3
Mean 47,870 70,276 4,666 61,665 76,802 76,591 1,682 121,065

•
Standard Deviation 7,687 12,116 5,191 6,165 10,652 4,666 331 31,783
Commercial Banking
Number
Mean
Standard Deviation
Payment & Settlements
41
43,083
6,918
3
63,248
10,905
28
4,199
4,672
20
55,499
5,549
41
69,121
9,586
37
68,932
4,199
170
1,514
298
2
108,959
28,605
Calculator
e.g.,
Number 37 3 26 18 37 34 153 2

0 1 2 3 4 Risk
Mean 38,774 56,923 3,779 49,949 62,209 62,039 1,363 98,063

•
Standard Deviation 6,226 9,814 4,205 4,994 8,628 3,779 268 25,744
Agency Services

Monte
Number 44 4 31 22 44 40 184 2
Mean 46,529 68,308 4,535 59,939 74,651 74,446 1,635 117,675
Standard Deviation 7,472 11,777 5,045 5,993 10,353 4,535 321 30,893
Asset Management
Number 40 3 28 20 40 36 165 2

Carlo
Mean 41,876 61,477 4,081 53,945 67,186 67,002 1,472 105,908

•
Standard Deviation 6,725 10,599 4,541 5,394 9,318 4,081 289 27,804
Retail Brokerage
Number 48 4 33 24 48 44 198 3

167,245 P Simulation
Mean 50,252 73,773 4,898 64,734 80,623 80,402 1,766 127,090
Standard Deviation 8069 12719 5449 6472 11182 4898 347 33365

Severity
Insurance
Number 43 4 30 21 43 39 179 2

99th
99th Percentile
Mean 45,226 66,395 4,408 58,260 72,561 72,362 1,589 114,381

142,456
Standard Deviation 7,262 11,447 4,904 5,825 10,063 4,408 312 30,028

Engine Mean
Mean percentile
of loss
Total
Number 435 36 302 217 435 399 1,812 24
Mean 45,653 67,021 4,450 58,810 73,245 73,044 1,604 115,459
Standard Deviation 7,331 11,555 4,950 5,880 10,158 4,450 315 30,311

123,345 Annual aggregate loss ($)

113,342
94,458
0-10 10- 20- 30- 40-
20 30 40 50

Under the BIS definition operational risk is defined as the unexpected loss

5 January 2005
 Industry viewpoint COSO
insistence of the BIS, loss data began fuelling to lead them out, they are likely to fall into
an entirely new and more scientific way of an even deeper abyss.
thinking about what came to be known as There are no shortcuts to developing a
operational risk management. It was the comprehensive framework for managing
analysis of this data and the issues operational risk. And one cannot get on the
subsequently raised that eventually led to the right track without confronting the difficult
development of modern operational risk issues head on. If an organisation’s
management as an objective discipline. operational risk management framework is
Some may still argue that COSO is useful

“
because it improves business process
ONE OF THE BIGGEST PROBLEMS WE
management; even if this is true, it should
FACE IN THE OPERATIONAL RISK MANAGEMENT
nevertheless be clear that business process
management is not operational risk management.
There are also those who speak of
operational risk management as independent
from operational risk measurement. In our
AREA IS THAT MANY OF THOSE PROFESSING TO
BE EXPERTS IN THIS FIELD ACTUALLY KNOW
VERY LITTLE ABOUT OPERATIONAL RISK
“
view measurement is an integral part of the MANAGEMENT IN ITS MODERN CONCEPTION
management process. After all, what is risk
management other than the mitigation of not founded on fundamentally sound
major risk in the most cost-effective way. It’s reasoning the entire programme will
difficult to see how one can accomplish this in eventually unravel at the seams. An ill-
a large organisation without reliable measures. conceived operational risk management
Basel II was introduced to encourage banks to programme is also likely to leave an
improve their operational risk management. But organisation vulnerable to major operational
following COSO does not improve operational losses. The damage from even one major loss
risk management; instead it promotes phantom could be far greater than the cost of
risk management and does more harm than establishing a state-of-the-art, integrated
good. Furthermore, any organisation that operational risk measurement-management
applies COSO-based risk-assessment to this end programme. Just think how little a very
will clearly be demonstrating to its regulators, simple global-early warning system would
to its investors and to the rating agencies that it have cost to build and maintain relative to
has not yet grasped even the most basic the lives lost and property damage that
understanding of operational risk management resulted from the recent Asian tsunami.
– ie, operational risk management is about The operational risk management industry
managing risk. In our view, far from meeting has been plagued by disinformation and
the standards of the advanced measurement methodology. The industry would be much
approach or even the standardised approach, a better served if instead of expressing the
COSO-based operational risk management personal opinions of the ‘experts’ it made an
framework may only just barely meet the effort to understand the issues. Without doing
minimum standards of the basic indicator so, it’s hard to see how anyone could become
approach – which has no standards at all!. qualified to address this challenging problem.
One of the biggest problems we face in the We certainly don’t pretend to have all the
operational risk management area is that answers, but we do think we have hit upon
many of those professing to be experts in this many of the right questions. It is important to
field actually know very little about recognise that one can never arrive at the right
operational risk management in its modern answers without probing the most important
conception. By continuing to espouse their issues. Only by analysing and re-analysing these
outmoded and impractical views on the issues can one begin to shed light on what may
subject these individuals are unknowingly be the right questions. Finding the answers is
doing more harm than good, as their flawed the easy part. Discovering what are the right
guidance is steering the industry in the questions is the major challenge. OpRisk
wrong direction. Based on the advice of these
individuals many organisations have invested Ali Samad-Khan is president of OpRisk Advisory. He has
millions of dollars implementing frameworks eight years’ experience in operational risk manage-
and software that they will soon discover have ment, having previously worked at Bankers Trust,
neither improved their management of OpRisk Analytics (which was acquired by SAS) and
operational risk nor achieved any level of BIS PricewaterhouseCoopers, where for over three years he
compliance. While it is easy to see how many headed the operational risk group within the New York
banks could have fallen into this conceptual FRM practice. He can be reached at
black-hole, if immediate steps are not taken ali.samad-khan@opriskadvisory.com

www.operationalriskonline.com 6