BY: David J.Litja: Measurement Errors

MEASUREMENT ERRORS
Copyright 2004 David J. Lilja

1
BY:
DAVID J.LITJA
Errors in Experimental Measurements
2
Sources of errors
Accuracy, precision, resolution
A mathematical model of errors
Confidence intervals
For means
For proportions
How many measurements are needed for desired
error?

Why do we need statistics?
1. Noise, noise, noise, noise, noise!

3
OK not really this type of noise
Why do we need statistics?
2. Aggregate data into
meaningful
information.

... = x
4
445 446 397 226
388 3445 188 1002
47762 432 54 12
98 345 2245 8839
77492 472 565 999
1 34 882 545 4022
827 572 597 364
What is a statistic?
5
A quantity that is computed from a sample [of
data].
Merriam-Webster
A single number used to summarize a larger
collection of values.
What are statistics?
6
A branch of mathematics dealing with the
collection, analysis, interpretation, and
presentation of masses of numerical data.
Merriam-Webster
We are most interested in analysis and
interpretation here.
Lies, damn lies, and statistics!

Goals
7
Provide intuitive conceptual background for some
standard statistical tools.
Draw meaningful conclusions in presence of noisy
measurements.
Allow you to correctly and intelligently apply techniques in
new situations.
Dont simply plug and crank from a formula.
Goals
8
Present techniques for aggregating large quantities
of data.
Obtain a big-picture view of your results.
Obtain new insights from complex measurement and
simulation results.
E.g. How does a new feature impact the overall
system?
Sources of Experimental Errors
Accuracy, precision, resolution
9
Experimental errors
10
Errors noise in measured values
Systematic errors
Result of an experimental mistake
Typically produce constant or slowly varying bias
Controlled through skill of experimenter
Examples
Temperature change causes clock drift
Forget to clear cache before timing run

Experimental errors
11
Random errors
Unpredictable, non-deterministic
Unbiased equal probability of increasing or decreasing
measured value
Result of
Limitations of measuring tool
Observer reading output of tool
Random processes within system
Typically cannot be controlled
Use statistical tools to characterize and quantify

Example: Quantization
Random error
12
Quantization error
Timer resolution
quantization error
Repeated measurements
X
Completely unpredictable
13
A Model of Errors
Error Measured
value
Probability
-E x E
+E x + E
14
A Model of Errors
Error 1 Error 2 Measured
value
Probability
-E -E x 2E
-E
+E x

+E -E x
+E +E x + 2E
15
A Model of Errors
Probability
0
0.1
0.2
0.3
0.4
0.5
0.6
x-E x x+E
Measured value
16
Probability of Obtaining a Specific Measured
Value
17
A Model of Errors
18
Pr(X=x
i
) = Pr(measure x
i
)
= number of paths from real value to x
i

Pr(X=x
i
) ~ binomial distribution
As number of error sources becomes large
n ,
Binomial Gaussian (Normal)
Thus, the bell curve
Frequency of Measuring Specific Values
19
Mean of measured values
True value
Resolution
Precision
Accuracy
Accuracy, Precision, Resolution
20
Systematic errors accuracy
How close mean of measured values is to true value
Random errors precision
Repeatability of measurements
Characteristics of tools resolution
Smallest increment between measured values
Quantifying Accuracy, Precision, Resolution
21
Accuracy
Hard to determine true accuracy
Relative to a predefined standard
E.g. definition of a second
Resolution
Dependent on tools
Precision
Quantify amount of imprecision using statistical tools
Confidence Interval for the Mean
22
c1 c2
1-
/2 /2
Normalize x
1
) (
deviation standard
mean
ts measuremen of number
/
n
1 i
2
1

= =
= =
=
=
=
n
x x
s
x x
n
n s
x x
z
i
n
i
i
23
Normalized z follows a Students t distribution
(n-1) degrees of freedom
Area left of c
2
= 1 /2
Tabulated values for t
24
c1 c2
1-
/2 /2
As n , normalized distribution becomes
Gaussian (normal)
25
c1 c2
1-
/2 /2
26
o
o
o
= s s
+ =
=

1 ) Pr(
Then,
2 1
1 ; 2 / 1 2
1 ; 2 / 1 1
c x c
n
s
t x c
n
s
t x c
n
n
An Example
Experiment Measured value
1 8.0 s
2 7.0 s
3 5.0 s
4 9.0 s
5 9.5 s
6 11.3 s
7 5.2 s
8 8.5 s
27
An Example (cont.)
14 . 2 deviation standard sample
94 . 7
1
= =
= =

=
s
n
x
x
n
i
i
28
An Example (cont.)
90% CI 90% chance actual value in interval
90% CI = 0.10
1 - /2 = 0.95
n = 8 7 degrees of freedom

29
c1 c2
1-
/2 /2
90% Confidence Interval
a
n 0.90 0.95 0.975

5 1.476 2.015 2.571
6 1.440 1.943 2.447
7 1.415 1.895 2.365

1.282 1.645 1.960
4 . 9
8
) 14 . 2 ( 895 . 1
94 . 7
5 . 6
8
) 14 . 2 ( 895 . 1
94 . 7
895 . 1
95 . 0 2 / 10 . 0 1 2 / 1
2
1
7 ; 95 . 0 1 ;
= + =
= =
= =
= = =
c
c
t t
a
n a
o
30
95% Confidence Interval
a
n 0.90 0.95 0.975

5 1.476 2.015 2.571
6 1.440 1.943 2.447
7 1.415 1.895 2.365

1.282 1.645 1.960
7 . 9
8
) 14 . 2 ( 365 . 2
94 . 7
1 . 6
8
) 14 . 2 ( 365 . 2
94 . 7
365 . 2
975 . 0 2 / 10 . 0 1 2 / 1
2
1
7 ; 975 . 0 1 ;
= + =
= =
= =
= = =
c
c
t t
a
n a
o
31
What does it mean?
32
90% CI = [6.5, 9.4]
90% chance real value is between 6.5, 9.4
95% CI = [6.1, 9.7]
95% chance real value is between 6.1, 9.7
Why is interval wider when we are more confident?
Higher Confidence Wider Interval?
33
6.5
9.4
90%
6.1 9.7
95%
Key Assumption
Measurement errors are
Normally distributed.
Is this true for most
measurements on real
computer systems?
34
c1 c2
1-
/2 /2
Key Assumption
35
Saved by the Central Limit Theorem
Sum of a large number of values from any
distribution will be Normally (Gaussian)
distributed.
What is a large number?
Typically assumed to be > 6 or 7.

How many measurements?
Width of interval inversely proportional to n
Want to minimize number of measurements
Find confidence interval for mean, such that:
Pr(actual mean in interval) = (1 )
| | x e x e c c ) 1 ( , ) 1 ( ) , (
2 1
+ =
36
37
2
2 / 1
2 / 1
2 / 1
2 1
) 1 ( ) , (
|
.
|
\
|
=
=
=
=
e x
s z
n
e x
n
s
z
n
s
z x
x e c c
o
o
o

38
But n depends on knowing mean and standard
deviation!
Estimate s with small number of measurements
Use this s to find n needed for desired interval width
39
Mean = 7.94 s
Standard deviation = 2.14 s
Want 90% confidence mean is within 7% of actual
mean.
40
Mean = 7.94 s
Standard deviation = 2.14 s
Want 90% confidence mean is within 7% of actual
mean.
= 0.90
(1-/2) = 0.95
Error = 3.5%
e = 0.035
9 . 212
) 94 . 7 ( 035 . 0
) 14 . 2 ( 895 . 1
2
2 / 1
=
|
|
.
|
\
|
=
|
.
|
\
|
=

e x
s z
n
o
213 measurements
90% chance true mean is within 3.5% interval
41
Proportions
42
p = Pr(success) in n trials of binomial experiment
Estimate proportion: p = m/n
m = number of successes
n = total number of trials
Proportions
43
n
p p
z p c
n
p p
z p c
) 1 (
) 1 (
2 / 1 2
2 / 1 1
+ =
o
o
Proportions
44
How much time does processor spend in OS?
Interrupt every 10 ms
Increment counters
n = number of interrupts
m = number of interrupts when PC within OS
Proportions
45
How much time does processor spend in OS?
Interrupt every 10 ms
Increment counters
n = number of interrupts
m = number of interrupts when PC within OS
Run for 1 minute
n = 6000
m = 658
Proportions
) 1176 . 0 , 1018 . 0 (
6000
) 1097 . 0 1 ( 1097 . 0
96 . 1 1097 . 0
) 1 (
) , (
2 / 1 2 1
=
n
p p
z p c c
o
95% confidence interval for proportion
So 95% certain processor spends 10.2-11.8% of its
time in OS
46
Number of measurements for proportions
47
2
2
2 / 1
2 / 1
2 / 1
) (
) 1 (
) 1 (
) 1 (
) 1 (
p e
p p z
n
n
p p
z p e
n
p p
z p p e
o
o
o
48
How long to run OS experiment?
Want 95% confidence
0.5%
49
How long to run OS experiment?
Want 95% confidence
0.5%
e = 0.005
p = 0.1097
| |
102 , 247 , 1
) 1097 . 0 ( 005 . 0
) 1097 . 0 1 )( 1097 . 0 ( ) 960 . 1 (
) (
) 1 (
2
2
2
2
2 / 1
=
=

p e
p p z
n
o
10 ms interrupts
3.46 hours
50
Important Points
51
Use statistics to
Deal with noisy measurements
Aggregate large amounts of data
Errors in measurements are due to:
Accuracy, precision, resolution of tools
Other sources of noise
Systematic, random errors
Important Points: Model errors with bell
curve
52
True value
Precision
Mean of measured values
Resolution
Accuracy
Important Points
53
Use confidence intervals to quantify precision
Confidence intervals for
Mean of n samples
Proportions
Confidence level
Pr(actual mean within computed interval)
Compute number of measurements needed for
desired interval width

BY: David J.Litja: Measurement Errors

Uploaded by

Copyright:

Available Formats

BY: David J.Litja: Measurement Errors

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BY: David J.Litja: Measurement Errors

Uploaded by

Copyright:

Available Formats

MEASUREMENT ERRORS

Copyright 2004 David J. Lilja

How many measurements?

You might also like