TAC How To Measure Anything

How to Measure Anything
Doug Hubbard Expert The Advisory Council www.TACadvisory.com
Copyright 2007 Hubbard Decision Research and The Advisory Council, Inc.
How to Measure Anything

I conducted 55 major risk/return analysis projects so far that included a variety of impossible measurements I found such a high need for measuring difficult things that I decided I had to write a book The book will be released in July 2007 with the publisher John Wiley & Sons This is a sneak preview of many of the methods in the book
A Few Examples
Org. EPA Problem
Assess value of Safe Drinking Water Information System SDWIS Forecast fuel use for the battle field
Findings
Identified and measured previously unnoticed risks & benefits Found factors that better correlated to actual fuel use
Benefit
$15 million improved NPV from reprioritizing functions
US Marine Corp
$50 million savings per year by reducing unnecessary battlefield inventory $2 million improved net benefit reprioritizing rollout
American Express
Analyze ROI of a dedicated test network for new applications
Determined key metric to prioritize replacements was downtime reduction
Source: Hubbard Decision Research
A Few More Examples

Risk of IT The risk of obsolescence The value of a human life The value of saving an endangered species The value of public health The value of IQ points lost by children exposed to methyl-mercury The value of better security The future demand for space tourism The value of better information The value of information availability Productivity and performance
Three Measurement Muses

Eratosthenes measured the Earths circumference to within 1% accuracy Enrico Fermi the physicist who used Fermi Questions to break down any uncertain quantity (and was the first to estimate the yield of the first atom bomb)
Emily Rosa the 11 yr old who was published in JAMA (youngest author ever) for her experiment that debunked therapeutic touch
Three Illusions of Intangibles

(The .com approach)
The perceived impossibility of measurement is an illusion caused by not understanding: the Concept of measurement the Object of measurement the Methods of measurement See my Everything Is Measurable article in CIO magazine
Before We Measure We Ask:

Why do you want to know? How much do you know now? What is the value to additional information?
Why Do You Want To Know?

The EPA needed to compute the ROI of the Safe Drinking Water Information System (SDWIS) Why?: To prioritize three specific upgrades We built a spreadsheet model that connected the expected effects of the system to relevant impacts
Source: Hubbard Decision Research Copyright 2007 Hubbard Decision Research and The Advisory Council, Inc.
Uncertainty, Risk & Measurement

Measuring Uncertainty, Risk and the Value of Information are closely related concepts, important measurements themselves, and precursors to most other measurements The Measurement Theory definition of measurement: A measurement is an observation that results in information (reduction of uncertainty) about a quantity. We model uncertainty statistically
ROI = 0% Risk of Negative ROI -25% 0% 25% 50% 75% Return on Investment (ROI) 100% 125%
Expected ROI
How Much Do You Know Now?

Most people can be taught how to subjectively assess their current state of uncertainty A small amount of training (three hours) can significantly improve calibration of estimates Comparisons of actual measures to original calibrated estimates show calibration works
10
Risk/ROI w/ Monte Carlo

Inputs
Administrative Cost Reduction
5% 10% 15%
% Improvement in Customer Retention

10% 20% 30%
Total Project Cost

$2 million $4 million $6 million
ROI
-50% 0% 50% 100%
11
What Is The Value of Measurement?

As a rule of thumb, the value of information is simply the cost of being wrong times the chance of being wrong The value of information on a range often just comes down to where the threshold is within the range.
B
Threshold: Below this point is losing money Relative Threshold (RT)=B/A
Worst Bound of the 90% CI; this is the undesirable end of the range
Best Bound of the 90% CI; this is the desirable end of the range
Source: How to Measure Anything: Finding the Value of Intangibles in Business
12
The IT(?) Measurement Inversion

Receives Most Attention
Costs
Least Relevant to Approval Decisions

Economic Relevance
Initial Development Costs Ongoing Maintenance/Training Costs
Receives Least Attention
See my article The IT Measurement Inversion in CIO magazine

Typical Attention
Benefits
A specific benefit (productivity, sales, etc.) Utilization (when usage starts and how quickly usage grows)
Chance of Cancellation
Most Relevant to Approval Decisions
13
Next Step: Observations

Now that we know what to measure and what its worth to measure, we can think of observations that would reduce uncertainty The value of the information limits what methods we should use, but we have a variety of methods available Take the Nike Method: Just Do It dont let imagined difficulties get in the way of starting observations
14
The Power of Sampling Methods

Several clever sampling methods exist that can measure more with less data than you might think Examples: estimating the population of fish in the ocean, estimating the number of tanks created by the Germans in WW II, extremely small samples, etc.

15
The Math-less Statistics Table

Measurement is based on observation and most observations are just samples Reducing your uncertainty with random samples is not made intuitive in most statistics texts This table makes computing a 90% confidence interval easy
16
Measuring to the Threshold

Measurements have value usually because there is some point where the quantity makes a difference Its often much harder to ask How much is X than Is X enough
17
The Simplest Method

Bayesian methods in statistics use new information to update prior knowledge Bayesian methods can be even more elaborate that other statistical methods BUT It turns out that calibrated people are already mostly subjectively Bayesian
18
Improving on Human Judgment

The Lens Model is one method used to improve on expert intuition The chart shows the reduction in error from this method on intuitive estimates In every case, this method equaled or bettered the judgment of experts
Business failures using financial ratios Psychology course grades IQ scores using Rorschach tests Student ratings of teaching effectiveness Mental illness using personality tests Changes in stock prices Graduate students grades Life-insurance salesrep performance Cancer patient life-expectancy 0% 5% 10% 15% 20% 25% 30% 35% 40%
Reduction in Errors

19
Prediction Markets
Simulated trading markets are a proven method of generating probabilities for uncertain events Research shows that it works even without purely monetary reward systems
Source: Servan-Schreiber, et. al. Electronic Markets, v 14-3, September 2004
20
Applied Information Economics

The basic AIE model was used in several of these examples where intangibles dominated the decision Even though it has been used on problems as different as IT portfolio prioritization Forecasting fuel for the battlefield the same basic process applies
Start: Define Decision(s) Build a decision model (Ongoing evaluations decisions and course corrections)
Select/Calibrate estimators
Add/update values in model
Compute Information Values
Collect/Analyze New Observations (PseudoBayesian or additional expert polling)
Do any Variables have a significant EVPI? Yes
No
Make Decision
Select/Design Measurement Instrument
Decompose the Variable (if possible)
21
Final Tips
Learn how to think about uncertainty, risk and information value in a quantitative way Assume its been measured before You have more data than you think and you need less data than you think Methods that reduce your uncertainty are more economical than many managers assume Dont let exception anxiety cause you to avoid any observations at all Just do it
22
More Information:
Several of my articles in CIO magazine, InformationWeek and other periodicals discuss measurement - especially the risk and value of IT. All of them are linked on www.hubbardresearch.com. The book will have its own web site at www.howtomeasureanything.com. The site will be online before the book is released in July. It will include detailed examples and spreadsheets mentioned in the book as well as discussion groups for readers. Contribute to The Measurement Challenge by sending us your examples of difficult measurements.
23

TAC How To Measure Anything

Uploaded by

Copyright:

Available Formats

TAC How To Measure Anything

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

TAC How To Measure Anything

Uploaded by

Copyright:

Available Formats

How to Measure Anything

Doug Hubbard Expert The Advisory Council www.TACadvisory.com

How to Measure Anything

Analyze ROI of a dedicated test network for new applications

Determined key metric to prioritize replacements was downtime reduction

Source: Hubbard Decision Research

A Few More Examples

Three Measurement Muses

Three Illusions of Intangibles

Before We Measure We Ask:

Why Do You Want To Know?

Uncertainty, Risk & Measurement

How Much Do You Know Now?

Risk/ROI w/ Monte Carlo

% Improvement in Customer Retention

Total Project Cost

What Is The Value of Measurement?

Source: How to Measure Anything: Finding the Value of Intangibles in Business

The IT(?) Measurement Inversion

Least Relevant to Approval Decisions

Initial Development Costs Ongoing Maintenance/Training Costs

Receives Least Attention

See my article The IT Measurement Inversion in CIO magazine

Next Step: Observations

The Power of Sampling Methods

Source: How to Measure Anything: Finding the Value of Intangibles in Business

The Math-less Statistics Table

Measuring to the Threshold

The Simplest Method

Improving on Human Judgment

Source: How to Measure Anything: Finding the Value of Intangibles in Business

Source: Servan-Schreiber, et. al. Electronic Markets, v 14-3, September 2004

Applied Information Economics

Add/update values in model

Compute Information Values

Collect/Analyze New Observations (PseudoBayesian or additional expert polling)

Do any Variables have a significant EVPI? Yes

Select/Design Measurement Instrument

Decompose the Variable (if possible)

Source: How to Measure Anything: Finding the Value of Intangibles in Business

You might also like