What Is Measurement Risk?

Henry A. Zumbrun
Morehouse Instrument Company, Inc.

Introduction a potentially negative consequence.” It further states that

“The focus of measurement quality assurance is to quantify
Imagine that a satellite is launched into space and and/or manage the ‘likelihood’ of incorrect measurement-
communications are intermittent. This happens because the based decisions. When doing so, there must be a balance
satellite is wobbling, which causes connection problems in between the level of effort involved in, and the risks
the receiver. The cause of the wobbling is identified: it is the resulting from, making an incorrect decision. In balancing
result of not using a calibration provider with a low enough the effort versus the risks, the decision (direct risk) and
uncertainty. The load cells used to measure the amount of the consequences (indirect risk) of the measurement must
fuel stored in the satellite must be highly accurate with very be considered.”
low uncertainties. However, if a calibration provider does ANSI/NCSLI Z540.3-2006 defines Measurement decision
not have the right measurement capability, the load cells risk as probability that an incorrect decision will result
will not be accurate enough to make the measurement. In from a measurement.
this case, the result is a wobbling satellite and significant
resources to fix the problem. What Does This Really Mean?
If the problem is not using a calibration provider with
an uncertainty adequate to perform the calibration, how All measurements have a percentage likelihood of calling
does one figure out how low is good enough? This article something good when it is bad, and something bad when it
answers this question by defining measurement risk, and is good. You might be familiar with the terms consumer’s
the role Test Uncertainty Ratios (T.U.R) play in reducing risk and producer’s risk. Consumer’s risk refers to the
measurement risk. possibility of a problem occurring in a consumer-oriented
product; a product that doesn’t meet quality standards
Understanding Measurement Risk passes undetected through a manufacturer’s quality control
system and enters the consumer market.
AS9100C defines risk as “[a]n undesirable situation or An example of this would be the batteries in the
circumstance that has both a likelihood of occurring and Samsung Note 7 phone. The batteries can potentially
overheat, causing the phone to catch
on fire. In this case, the faulty battery/
charging system of the phone device
was approved through the quality
control process of the manufacturer,
which was a ‘false accept decision.’ If
you owned one of these phones, there
was a risk of fire and potential damage
and injury.
In metrological terms, consumer’s
risk is like the false accept risk, or
Probability of False Accept (PFA).
The biggest difference is that in the
metrology field, the false accept risk
is usually limited to a maximum of 2
percent. In cases where the estimation
of this probability is not feasible, there
is a requirement for a Test Uncertainty
Ratio (TUR) to be 4:1 or greater to
ensure lowering the PFA to a low risk
Figure 1. Graph showing the measurement risk which is the Probability of False Accept
level.
(PFA).

Apr • May • Jun 2017 27 Cal Lab: The International Journal of Metrology
 What Is Measurement Risk?
Henry A. Zumbrun

So, what does this mean for a metrology laboratory? Uncertainty are as follows:
It means that any lab making a statement of compliance,
calling an instrument “in tolerance,” must consider Tolerance
___________________
TUR =   
​    ​
measurement uncertainty and properly calculate T.U.R. Expanded Uncertainty
considering the location of the measurement. In simplistic
(USL − LSL)
terms, T.U.R. = Tolerance Required/Uncertainty of the TUR = ___________
​ 
4 * u    ​ 
Measurement (at a 95% confidence interval). If the
Uncertainty of the Measurement is not less than the where:
tolerance required, there will be a significant risk of false TUR = Test Uncertainty Ratio,
accept. In simplistic terms, a TUR that produces less than USL = Upper Specification Limit,
+/- 2 % upper and lower risk would be required to ensure LSL = Lower Specification Limit, and
the measurement is valid. u = standard uncertainty.
Keys to lowering measurement risk include having Note: We are using 4 assuming k=2, the proper formula
your calibration provider replicate how the instrument is would be 2 times the actual k value is for a 95 % confidence
used in the field, having competent technicians, using the interval.
right equipment, and lowering overall uncertainties by
the calibration provider. There is quite a bit of difference
between force measurement labs with CMCs of 0.1 percent,
The Calculation of TUR for Tolerances
0.05 percent, 0.02 percent, 0.01 percent, 0.005 percent and
0.002 percent of applied force. Not using the laboratory • ((Upper Specification Limit - Lower Specification
with the right capability to meet your requirements is like Limit))/(4 * Standard Uncertainty)
using a ruler to calibrate a gauge block. • Combined Uncertainty (u) – The square root of
Table 1 shows the Test Uncertainty Ratios (TUR) that the sum of the squares of all the input quantity
force calibration labs with different calibration capabilities uncertainty components.
can provide for various levels of required tolerances. The __________________________

√ (  ) (  ) (  )
far-left column represents the calibration standard required CMC 2 ______ Res 2 ____ Rep 2
u =​ ​​ ______
​     ​  ​​ ​+ ​​ ​     
3.464   ​  ​​ ​+ ​​ ​  1    ​  ​​ ​ ​
for force measurements. Deadweight primary standards k
are often required to achieve CMCs of better than 0.01
• CMC = Calibration and Measurement Capability.
% of applied force. A high-end load cell calibrated by
This should be found on the calibration report.
deadweights would be required to achieve CMCs of better
than 0.05 %. This table indicates the best TUR that the labs • Res = This is the resolution of the Unit Under Test
can provide for the same load cell at similar conditions. Per (UUT) The divisor for resolution will either be
this table, only calibration labs with CMCs around 0.02 % 3.464 or 1.732 (depending on how the UUT least
or better can calibrate devices with a tolerance of 0.1 %. significant digit resolves).
They may still need to adjust the device to read closer to • Rep = Repeatability of the Unit Under Test (UUT).
the nominal value. We will discuss guard banding later. Repeatability of UUT must be used if repeatability
The table was derived from TUR and uncertainty studies were not previously accounted for in the
formulas found in JCGM 100:2008 and ANSI/NCSLI CMC. If accounted for in the CMC, this would
Z540.3-2006. The formulas used to determine TUR and not be required.

Table 1. TUR Table

Cal Lab: The International Journal of Metrology 28 Apr • May • Jun 2017
What Is Measurement Risk?
Henry A. Zumbrun

• Expanded Uncertainty - Typically 2 times the Examples of Calculating Measurement Risk

standard uncertainty. However, the appropriate with Guard Banding
k value should be used to ensure a coverage
probability of 95 %, based on the effective degrees Assume we are testing a load cell at 10,000 lbf force. The
of freedom using the Welch Satterthwaite formula. accuracy specification is 0.1 % of reading (or +/- 10 lbf at this
force), and the measured value was 9990. Is the device in
Is Your Calibration Provider Reporting Pass/ tolerance? After all, the calibration laboratory applied 10,000
Fail Criteria Properly? lbf and the unit under test (UUT) read 9990. The bias is -10 lbf
and the device meets its accuracy specification (accept decision
If the calibration provider is accredited, it needs to follow without taking the uncertainty of measurement into account).
the requirements per ISO/IEC 17025. ISO/IEC 17025:2005 The report is issued and the end user is happy. However,
Clause 5.10.4.2 states that “When statements of compliance the problem is that the end user should not be happy. If
are made, the uncertainty of measurement shall be taken the calibration and measurement capability (CMC) of the
into account.” calibration laboratory using a specific reference standard
This translates to minimizing the Probability of False was not considered, the end user will not know whether the
Accept (PFA) by applying a guard banding method. ANSI/ device meets the accuracy specification required. Basically,
NCSLI Z540.3 -2006 Handbook discusses guard banding this measurement was passed based on the assumption
in section 3.3. Section 3.3 paragraph 2 states “As used in that the calibration providers reference was perfect and
the National Standard, a guard band is used to change the they applied exactly 10,000 lbf to the load cell. However,
criteria for making a measurement decision, such as pass or no measurements are perfect. That is why we estimate the
fail, from some tolerance or specification limits to achieve a uncertainty of measurement to quantify this “imperfection of
defined objective, such as a 2 % probability of false accept. the measurement.” This is a false assumption which neglected
The offset may either be added to or subtracted from the the uncertainty in the calibration provider’s measurement.
decision value to achieve this objective.” Let us assume that the standard uncertainty was calculated
at 6.5 lbf for k=1. In Figure 2, the item being calibrated would
normally be considered
“in tolerance” by a large
percentage of calibration
laboratories since the
accuracy specification is
0.1 % of reading or +/- 10
lbf and the measured value
was within the accuracy
specification at 9990 lbf.
There is a 50.1 % chance
of the calibration being
accepted when it is not in
tolerance.
Figure 2. Figure 3 shows the risk
when the measured value
of the UUT reads 10,000
lbf. In this scenario, the
bias or measurement error
is 0. However, there is still
a 12.39 % chance that the
UUT is not “in tolerance.”
Simply put, there is too
much risk. We need to lower
the standard uncertainty
to reduce the risk. Note
that the TUR remains the
same since it is a ratio not
dependent on the location
of the measurement.
Figure 3.

Apr • May • Jun 2017 29 Cal Lab: The International Journal of Metrology
 What Is Measurement Risk?
Henry A. Zumbrun

Figure 4.

How to Lower the Risk (PFA) By Lowering This road is not more difficult; it’s just different from the
the Uncertainty current way you may be doing things. Choosing to consider
the impact of not doing things right—and making the
decision to select the best calibration provider—will make
1. Use better equipment with a lower resolution and/ all the difference. The rest is just putting formulas in place
or better repeatability; e.g., higher quality load cell to report and know your measurement risk.
for force measurement.
2. Use a better calibration provider with a Calibration References
and Measurement Capability (CMC) low enough
to reduce the measurement risk.
[1] JCGM 100:2008 Evaluation of measurement data — Guide
3. Pay attention to the uncertainty values listed in
to the expression of uncertainty in measurement, BIPM.
the calibration report issued by your calibration
provider. Make sure to get proper T.U.R. values [2] ILAC P14:01/2013 Policy For Uncertainty in Calibration.
for every measurement point (but pay attention
to the location of the measurement). [3] ANSI/NCSLI Z540.3-2006 Requirements for the Calibra-
The last graph (Figure 4) shows the same test instrument tion of Measuring and Test Equipment.
with a lower Standard Uncertainty. This was a real scenario [4] ISO/IEC 17025:2005: General requirements for the compe-
where an instrument was modified from a 10 lbf resolution
tence of testing and calibration laboratories.
to a 2 lbf resolution. The total risk is now 0 and the device
will be “in tolerance” with less than 2 % total risk from [5] AS9100C Quality Management Systems - Requirements
reading of 9,996 through 10,006 lbf. There are several
for Aviation, Space, and Defense Organizations.
acceptable methods for applying a guard band to obtain
what the measured value needs to be in order to maintain
less than 2 % total risk.
These graphs comply with Method 5: Guard Bands Henry A. Zumbrun (hzumbrun@morehouse.com),
Based on Expanded Uncertainty in the ANSI/NCSLI Z540.3 Morehouse Instrument Company, Inc., York, Pennsylvania.
Handbook and is described in ISO 14253-1, and included in
ILAC G8, and various other guidance documents.
After reading this paper, you may be standing at a
crossroads and wondering if any of this extra work is
necessary. To the left is the same rough path you’ve
been travelling all along. This is the path that says, “If
it’s not broken, why fix it?” You might be thinking that
measurement risk has not been an issue before, or you’ll
just wait until an auditor questions you about it (or there is
a train wreck). Yet, to the right is the road that fewer people
realize will help solve their measurement problems today.

Cal Lab: The International Journal of Metrology 30 Apr • May • Jun 2017