Clinical CMN

Microbiology
Stay Current...
Stay Informed.

N E W S L E T T E R

CMN Sigma Metrics for Assessing Accuracy of

Vol. 37, No. 13
July 1, 2015
Molecular Testing
www.cmnewsletter.com Sten Westgard, M.S.1 and Danijela Lucic, Ph.D.,2 1Westgard, Inc., Madison, Wisconsin,
2
Department of Scientific Affairs, Abbott Molecular, Inc., Des Plaines, Illinois
I N TH I S I S S U E Abstract
103 Sigma Metrics for For any diagnostic testing, the bar for accuracy is high, for good reason. Disease status and therapy deci-
Assessing Accuracy of sions are often defined by a single positive or negative result. In the case of molecular viral load testing
Molecular Testing of patients with hepatitis or human immunodeficiency virus infections, expensive antiviral therapies
are continued or terminated based on laboratory results. If these results are not accurate, patients may
be subjected to additional testing or to treatments, which can add expense, worry, and in some cases
medical harms. Viral load cutoff points for therapeutic decisions have been recently lowered; therefore,
test result accuracy near the assay’s lower limit of quantitation is even more important today than in
the past. How can molecular diagnostic laboratories improve the assessment of their viral load assay’s
accuracy? Answers can be found in the statistical assessment of results derived from testing quality
control material. The statistical tools are collectively called Sigma metrics. This review will detail the
aspects of Sigma metrics that relate to viral load testing, and also review recent literature for common
viral load testing as it relates to Sigma metrics.

Why Do Quality Metrics Matter? We contend that proper patient care can be built
The major assumption physicians make prior only on a foundation of quality, which is sup-
to their clinical decisions is that the laboratory’s ported by metrics and analytics. Regardless of
test results are valid - that there is no medically the disease state, assay quality must begin with
important error that obscures the true result, assessment of an assay’s analytical performance
generating either a false-positive or a false-neg- and extend to the use of its results at the relevant
ative result. Laboratory medicine embraces the clinical (medical) decision points. In this review,
paramount importance of quality. With a long we present analytic tools that one can use to criti-
history of quality and accuracy for diagnostic cally assess an assay’s performance by analyzing
testing, it is easy to assume that a test result has results from method verification data or quality
the appropriate accuracy, but how can we ensure control data.
that the necessary accuracy is being achieved? How Do We Know That When We Achieve
How can we ensure that medical decisions are the Proper Level of Quality?
not made based on faulty assumptions? We know
The core requirement of any method used for
inherently that patient diagnosis, and in some
quality assessing is the ability to numerically
cases, treatment may be endangered by poor test
define what we consider good performance and
Corresponding author:
performance, regardless of the skill of the clini-
acceptable quality; therefore, we must also define
Danijela Lucic, Ph.D., Abbott cian or the laboratorian, but do we know how
unacceptable quality and poor performance, i.e.,
Molecular, Inc., 1350 East Touhy to analyze our quality control data to prove the
we must define an unacceptable error, i.e., a defect
Ave., Des Plaines, IL 60018. assay is functioning properly? We certainly col-
Tel.: 224-361-7124. Fax: in the method’s performance. For many diagnostic
lect quality control data, and review it, but do we
224-234-7707. E-mail: tests, the definition of what constitutes acceptable
critically assess the results?
danijela.lucic@abbott.com quality is implicit, but not often clearly articu-

Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier 103

lated in an analytical context. For viral loads, such as those for When adapted for clinical laboratories, the “Six” in Six Sigma
quantitation of human immunodeficiency virus type 1 (HIV-1) comes from the idea that six SDs of process variation must fit
or hepatitis C virus (HCV), we can define a defect or an error as between the true value of a test result and the defined tolerance
any instance in which a patient result is misclassified as it related limit (16,17). When variation is limited to this degree of precision
to medical decisions. In that context, there can be two kinds of (Six Sigma), the process generates only approximately 3.4 DPMO,
defects; one occurs either when a positive result is misclassified on what is called the short-term Sigma scale. When a process
as negative based on viral load, the other occurs when a normal, achieves Five Sigma on the short-term scale, approximately 233
disease-free patient result is misclassified as positive or diseased. DPM occur. With Four Sigma, the number rises to 6,210 DPM,
Clearly, we want to avoid false-positive and false-negative results, and with Three Sigma, it rises again to 66,807. The rapid rise in
as both can impact medical outcomes. For example, a patient who defects observed as the Sigma metric declines helps to explain
is allowed to believe their virus is truly suppressed or eradicated why Six Sigma is considered the ideal performance standard and
because of an inaccurate test result may not receive the treatment Three Sigma is typically considered the minimum acceptable
they need. Conversely, a patient who is truly healthy or cured, performance level for a process or assay. As a side note, to most
but issued a positive viral load report, would potentially receive laboratorians, there is only one Six Sigma scale; however, there are
unnecessary medical treatment and additional testing. Both sce- actually two scales. The first is a short-term scale, and the second
narios are financially and clinically unacceptable outcomes (1-7). is a long-term scale (Table 1a and 1b, respectively).

The history of Six Sigma is built on the development of the

A “Traditional” Approach to Quality: Six Sigma short-term scale, which is 1.5 Sigma higher than the long-term
The quality management techniques collectively known as “Six scale. That is, a 4.5 Sigma process on the long-term scale is the
Sigma” have been practiced in health care for several decades, and equivalent of a Six-Sigma process on the short-term scale. The
for even longer in manufacturing, business, and industry (3,5,8- difference occurred when Motorola created the scales, they justi-
17). The implementation of Sigma metrics has mostly occurred fied a 1.5 Sigma difference based on observations that a process
in laboratory subspecialties, such as chemistry, hematology, and will vary and drift over the long term by about 1.5 Sigma. When
immunology. processes are benchmarked by the counting of defects, the Sigma
metric is typically expressed on the short-term scale, outside of
The core concept of Six Sigma is to identify defects (false nega-
the diagnostic laboratory.
tive or false positive reactions) and then reduce or eliminate as
many of those defects as possible until a nearly defect-free opera- Because of the unique nature of analytical laboratory testing,
tion is achieved. The Six-Sigma technique can be applied to any where it is can be difficult to identify and count a defect in real
process, if you can define what the defects are, detect them when time, the Sigma metric is instead calculated using measures of the
they occur, and implement improvements that reduce or eliminate bias and the imprecision of the testing method. For laboratories,
the occurrence of those defects. this Sigma metric calculation is associated with the long-term
Mathematically speaking, a defect occurs whenever a process
outcome (for example, a viral load result) deviates beyond a pre-
defined tolerance limit. That is to say, there is a numerical value Table 1a. Short-term Sigma scale
that the test process should produce (sometimes called the “true
Sigma Level Defects per Million Performance
value”) and there is defined amount of variation that is consid-
1 Sigma 690,000 Unacceptable
ered by experts to be acceptable. The acceptable variation bounds
the true value on either side of the true value (e.g., a mean, or an 2 Sigma 308,000 Poor
International Standard, etc.). For example an average, or mean, is 3 Sigma 66,800 Marginal
generally reported in the literature as the mean value plus or minus 4 Sigma 6,210 Good
the calculated standard deviation (SD). In this example, the SD is 5 Sigma 230 Excellent
a common measure of variation around the mean.
6 Sigma <3.4 World Class
For diagnostic testing, it is common for laboratorians, clinicians,
or statisticians to place a reasonable limit on the amount of varia- Table 1b. Long-term Sigma scale
tion that can be accepted or tolerated. The boundary of that
Sigma Level Defects per Million Performance
acceptable variation is called the tolerance limit, which is generally
specific to the assay, the disease, or the therapeutic decision points. 1 Sigma 158,655 Unacceptable
Therefore, whenever the process variation exceeds the tolerance 2 Sigma 22,750 Poor
limit, a process defect is said to occur (16,17). In Sigma metrics, 3 Sigma 1,350 Marginal
the number of defects that occur over time is reported on a scale 4 Sigma 32 Good
of defects per million (DPM) or defects per million opportunities
5 Sigma <3.4 Excellent
(DPMO). The variation might be defined by the SD, the coeffi-
cient of variation, or the Sigma metric. 6 Sigma <3.4 World Class

104 Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier

scale, thus it is a more conservative approach; it does not build in to the longitudinal disease state of the patient.. Simple observation
the 1.5 Sigma increase to the benchmark. is not sufficient to identify all defects in viral load assays.

While the reduction in defects is a reward in itself, there are other For analytical methods, like viral load assays, an alternative
obvious benefits to achieving Four-, Five-, and Six-Sigma assay approach must be deployed in order to calculate the Sigma met-
performance. For manufacturing and industry, achieving Six Sigma ric. Rather than count defects, we instead calculate the expected
means operating at the highest efficiency, process reliability, and Sigma metric through an equation that combines the impacts of
customer satisfaction, and ultimately, all of these effects generate imprecision and inaccuracy. The Sigma metric equation (16,17)
the maximum product quality and profit. In contrast, processes is as follows: Sigma metric = (TEa < bias)/CV. In this equation,
operating below Three Sigma are typically the most error-prone, the TEa, bias, and coefficient of variation (%CV) are all expressed
resource-consuming, difficult-to-maintain, and expensive pro- as percentages and the absolute value of the bias is used. For any
cesses in an organization. Outside of health care, processes oper- molecular method, as long as all the variables are expressed in
ating below Three Sigma are usually considered too unstable either logarithmic or non-logarithmic numerical scales, it is pos-
for routine operation because they generate too much re-work. sible to calculate a Sigma metric.
These are processes that must be radically improved, redesigned,
or, ultimately, replaced.
For viral loads, the TEa value is established based on clinical
Similarly, within health care laboratories, test methods that oper- guidelines, so the percent TEa reflects a medically important
ate below Three Sigma typically generate significant re-work change (i.e., 0.5 log copies/ml in and HIV viral load). The %CV
in the form of repeated controls, repeated calibrations, trouble- and bias values are obtained from repeated measurement of labora-
shooting, and even a need for repeated testing of the patient in tory control material, used to assess precision during the method
order to determine a diagnosis. Indeed, in some cases, diagnostic verification or by analyzing longitudinal quality control data. The
methods performing significantly below Three-Sigma have been %CV and bias values can also be obtained from comparisons with
recalled from the market. But the concepts of Six Sigma are rela- peer group surveys or proficiency testing surveys.
tively new to molecular diagnostic laboratories and are not yet
commonly calculated. Awareness of the Sigma metrics should The Sigma metric can be calculated with the variables expressed in
help molecular laboratories better define assay performance and unit-based measurements, as long as all the terms of the equation
improve overall quality. are kept consistent. The graph in Fig. 1 is a visual explanation of
the Sigma metric equation, showing how both bias and impreci-
'H¿QLQJ4XDOLW\5HTXLUHPHQWVIRU6L[6LJPD3HUIRUPDQFH sion affect the distribution of test results. Whenever the curve of
the distribution exceeds the TEa (the lines drawn on either side of
Most molecular laboratories are not fully accustomed to discuss-
ing the “tolerance limits” of our test methods. Laboratorians are
far more comfortable discussing a similar concept, known as the
allowable total error (TEa). The TEa, the net analytical error is
defined as a combination of method imprecision (random error)
and method inaccuracy (systematic error, or bias) (16,17). When
a test method exceeds the TEa, the method or process begins to
generate defects (result errors), which could be classified as either
false positive or false negative results. When we want to prevent
our analytical method from exceeding the Tea, we can rely on
Sigma metrics. If we can achieve Six-Sigma performance, in other
words, maintain our imprecision at approximately less than 1/6 of
the TEa, we keep our defect rate below 3.4 DPM.

For analytical testing processes, an additional challenge is the cal-

culation of the Sigma metric itself. For example, outside the labo-
ratory, the Sigma metric could be is determined by simply counting
the defects produced in a manufacturing process. Given a sample Figure 1. Visual representation of the Sigma metric equation. The two
size of x and a number of observed defects of y, the y/x ratio is cal- curves of the test method distribution show two different scenarios:
WKHÀUVWZKHUHWKHGLVWULEXWLRQLVFHQWHUHGRQWKHWUXHYDOXHGLUHFWO\
culated and converted into DPM, and the DPM is then matched in the center, and has imprecision reduced to 1/6 of the TEa, is expected
to the Sigma metric. This is easy to do for processes where the to generate less than 3.4 DPMO; the second distribution curve is not
defect is obvious and can be counted in real-time, for example, centered on the true value but instead is biased to the right by a
missing specimens or excessive turnaround time. For analytical systematic error and is further affected by a larger imprecision that
spreads out the distribution curve. As a result of the combination of
methods, it is more difficult to determine whether a test result is bias and imprecision, part of this curve exceeds the tolerance limit
a defect without doing parallel testing against a reference method 7(DDQGDOORIWKHDUHDXQGHUWKHFXUYHDIWHUWKDWGHÀQHVWKHQXPEHU
or performing retrospective clinical chart review with correlation of defects expected by the method.

Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier 105

the true value), the area remaining under the curve represents the
number of expected defects.
While the Sigma metric can be calculated, it can also be visu-
ally displayed using a method decision chart (Fig. 2). This chart
plots the performance of a test method as a combination of the
observed imprecision and bias, specifically, using imprecision as
the x coordinate and bias as the y coordinate. The diagonal lines on
the chart represent different Sigma zones, from world class qual-
ity performance (better than Six Sigma) near the graph’s origin to
excellent performance (Five Sigma), to good performance (Four
Sigma), and so on, until the last zone, the upper right quadrant of
the graph, represents less than Two-Sigma performance, which
is considered unacceptable, unstable, and unreliable. The visual
simplicity of the graph shows the relationship between the various
Sigma conditions: the closer to the origin, the fewer defects and
the more confidence the laboratory and clinicians can place in the
test result. Conversely, as imprecision and bias grow, they progres-
sively impair a method’s performance, generating more defects and
confusing the clinician instead of confirming a diagnosis.
Figure 2. Method decision chart; visual representation of the Sigma
The Challenge of Adapting Six Sigma to Molecular Methods metric.

For most laboratory methods, broad classification of positive and

negative results can be easily defined; however, when one looks 7KH5HDO:RUOG,PSRUWDQFHRI6L[6LJPD4XDOLW\IRU+,9
more closely at the analytical performance, precision becomes 9LUDO/RDGV
more challenging to describe. Typically, clinically significant
For HIV and HCV testing, there are test utilization scenarios
cutoffs for HIV and HCV load tests are reported in a logarith-
with immediate, profound clinical and financial consequences.
mic scale or in international units (18-21). It therefore requires a
Therapies may be terminated or extended and patients may receive
change of scale in order to import and adapt the traditional tools
more or less care based on viral load test results. The clinicians
of statistical quality control. It becomes difficult to define quality
and patients assume that the quality of the assay allows for the
if the laboratory is not certain which cutoffs are the most relevant.
correct answer to be delivered with just one test, but given the
Is it sufficient to ensure that the typical healthy patient does not
new medical decision points expected for some viral load assays
exceed the plus or minus 0.5-log-cps/mL limit in variability? Or is
that assumption that should be confirmed through objective and
it better to take the more conservative approach and use the high-
rigorous analysis such as Sigma metrics.
est “negative” value and make sure that the value never exceeds
0.5-log-cps/mL difference? In contrast, is it more appropriate to Current therapy guidelines (Department of Health and Human
choose the lowest positive value and demand that the low posi- Services [DHHS]) for HIV-1 state that virologic failure occurs
tive never fall below the 0.5-log-cps/mL difference? In one rec- when the viral load exceeds 200 cps/ml, and as such, a physician
ommended approach, laboratories can choose to adopt the more would need to confirm this change in the viral load with a subse-
demanding quality requirements at either end of the quantitative quent viral load measurement (22-25). An abrupt rise in the viral
spectrum, i.e., that the highest negative patient never exceeds the load may simply indicate a “viral blip”—a temporary rise that will
cutoff, and likewise, the lowest positive patient never falls below resolve itself or an actual change in the patient’s status (virologic
the cutoff (16,17). failure). Here, we review two different published examples of viral
load assay performance and how imprecision can impact patient
Most importantly, a growing number of medical therapies, the
management and therapy.
historical range of acceptable variability (0.5-log-cps/mL) is no
longer the most important decision point used by clinicians for Naeth et al. (26) evaluated patient samples at 40, 80, and 90 cps/
disease management. For many treatment regimens, a far lower ml (Fig. 3a). In this publication, one real-time assay’s precision
cutoff is used, one that defines the success of a drug therapy as the (Abbott Molecular) ranged from 26 to 32% CV, while the compar-
achievement of eradication or near eradication of the viral load ator assay (Roche Molecular Diagnsotics) precision ranged from
(1,2,6). In these cases, the numerical value reported is often the 37 to 59% CV. One can see that at 90 cps/ml, latter assay’s values
actual unit measured, not a number converted to the logarithmic could range from 56 to 200 cps/ml, while the former ranges from
scale. To determine the quality of method performance in those 70 to 100 cps/ml. Using the Neath study data and applying the
scenarios, it will be more appropriate to use the actual units to Sigma metric, authors were able to ascertain how likely it would
define the TEa. Thus, we may express the TEa for a test on either be to identify a real change in the viral load within the range of
the logarithmic or non-logarithmic scale, depending on the clini- 40 to 200 cps/ml. This viral load change from 40 to 200 cps/ml
cal decision level we are interested in assessing. (0.7 log copies/ml) is clinically significant, as it is greater than the

106 Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier

0.5 log10 unit currently required by the DHHS guidelines. In this at densities of 12, 25, 50, and 100 cps/ml (Fig. 4a). In this study,
example, this change in the viral load can be used as the TEa for one assay’s precision ranged from 33 to 68% CV, while compara-
both Sigma metric evaluations. For their calculations, the bias tor methods precision ranged from 33 to 83% CV. Similar to the
value was held constant for both methods due to the original value previous example, after applying the Sigma metric, authors are
assignment. CVs, which were used to assess viral load changes, able to ascertain the ability of an assay to identify a significant
were 26% and 51%, respectively. Sigma analysis demonstrated change in the viral load from 25 to 400 cps/ml (a 1.2-log-copies/
that within this clinical decision interval (40 to 200 cps/ml) one ml change). This change in the viral load represents a larger shift
assay performance achieves a Six-Sigma level while the compara-
than DHHS considers virologic failure. Subsequently, it is impor-
tor performance achieved only the Three-Sigma level (Fig. 3b).
tant to understand the assays’ performance within the medically
Another example of viral load performance at clinical cutoff is important region(s) of the dynamic range and compare DPMOs
the evaluation by Ruelle et al. (27). This study evaluated samples within those regions. Similar to the Neath analysis, the Ruelle

A B

Figure 3. (a) Neath et al. analytical analysis at 40, 80, and 90 cps/ml. (b) Method decision chart for viral load change from 40 to 200 cps/ml.

A B

Figure 4. (a) Ruelle et al. analytical analysis at 12, 25, 50, 100 cps/ml. (b) Method decision chart for viral load change from 40 to 400 cps/ml.

Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier 107

bias value was held constant for both methods. With the expan- of Quantitation (LOQ) to 50 IU/ml and the bias is held constant.
sion to the low end of the viral load range, the % CVs were 41% One can see that the assay performance ranged from Six Sigma
and 83%, respectively. Sigma analysis demonstrated that the com- to Three Sigma (Fig. 5b). What does this mean clinically and for
parator method achieved a Four-Sigma performance while Abbott patients? It means that use of the Six Sigma method could benefit
Molecular assay maintained Six-Sigma performance (Fig. 4b). patients, as it can document a more accurate assessment of sus-
tained virologic response clearance of HCV.
Identification of HIV-1 virological failure is critical, as patients
typically visit physicians only twice per year. A medically important
Similar precision performance was observed by Kessler et al. when
error in a test result for viral load could lead to a patient remain-
the World Health Organization International Standard was used to
ing on failing therapy, which could lead to resistance and could
evaluate precision at 100, 50 and 25 IU/mL (29). When variance
increase the possibility of transmission. In contrast, utilizing a Six
was converted to coefficient of variation, this study demonstrated
Sigma assay would avoid obscuring a medically important error
that at 25 IU/mL, the Abbott assay’s %CV was 36% while the
within the the low end of viral density (aka noise). Assays that can
comparator’s CV was 95% (data not shown). Applying similar TEa
achieve Six Sigma will by their nature have fewer false positives
as in the previous example, the assay performed at the 5-Sigma
and reduce unnecessary repeat testing.
level while the comparator performed at the 2-Sigma level. Again,
7KH5HDO:RUOG,PSRUWDQFHRI6L[6LJPD4XDOLW\³+&9 the Five Sigma can more accurately detect changes in viral load,
which confirm sustained virologic response, assess viral clearance
In a study by Wiesmann et al. (28), HCV patients were monitored
of HCV, or indicate virologic failure.
for a 2-log drop from baseline to ensure therapeutic response. As
with HIV load methods, the diagnostic assay precision can directly
Using the Kessler data, the comparator assay would not meet
impact whether or not a patient remains on HCV therapy. As
the sigma criteria for acceptable performance, given the current
therapies have become effective, the clinical decision points have
decisions for TEa limits. One cannot broaden the TEa to accept
migrated to the low end of the dynamic range, where accuracy
more variability given that the medical impact of small errors in
and precision represent challenges for all real-time PCR assays.
viral load results, including those at the medical decision point is
Current HCV treatment guidelines state that sustained virologic not inconsequential. Under the current medical cutoff points a
response at the end of treatment is achieved if the viral load is <25 larger amount of inherent imprecision can only be mitigated by
IU/ml. Wiesmann et al. evaluated HCV assay reproducibility at 25 testing duplicate triplicate samples and using the average value as
IU/ml using two commercial HCV assays and demonstrated that the viral load results. While this approach will improve accuracy
precision ranged between 23% CV and 51% CV (Fig. 5a). Using of methods with high inherent error, if also adds cost to the pro-
the analytical performance at 25 IU/ml, the authors applied Sigma cess. Selection of the most accurate and precise method to detect
analysis, where TE is the change in viral load from Lower Limit changes in viral load will more readily confirm sustained virologic

A B

Figure 5. (a) Clinical samples at 25 IU/ml were run 10 times. The mean CVs were 26% and 65% for ART and CAP/CTM, respectively. (b) Method
decision chart for viral load change from LOQ to 50 IU/ml.

108 Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier

response, assess viral clearance, identify virologic failure, and limit performance in the laboratory and determining its most effective
overall testing costs. use in diagnostic and treatment pathways.

Discussion Disclosure
Molecular assays have been utilized to manage HIV-1 and HCV Danijela Lucic is an employee of Abbott Molecular Inc.; Sten
therapy response for decades. Differences in accuracy and pre- Westgard is a consultant for Abbott Inc.
cision at clinical decision points where treatment decisions are
5HIHUHQFHV
being made and patient management is impacted have become a
point of laboratory discussion. Historically, laboratories have chal- 1. Ambrosioni, J. et al. HIV treatment outcomes in Europe and North
lenged assay performance only across the full linear range (overall America: what can we learn from the differences? Expert Rev. Anti-
Infect. Ther. 12:523-526.
bias); however, there is a shift towards assessing assay performance
2. Coppola, N. et al. 2015. Treatment of chronic hepatitis C in patients
near or at the clinical decision points. This strategy proved useful with HIV/HCV coinfection. World J. Virol. 4:1-12.
when the first generation of direct-acting antivirals was approved 3. Easterbrook, P.J. et al. 2014. The role of mathematical modelling in
for HCV treatment. Assay sensitivity and precision were critical the development of recommendations in the 2013 WHO consolidated
for appropriate HCV therapy management and duration. The antiretroviral therapy guidelines. AIDS 28(Suppl. 1):S85-S92.
concept is also useful for HIV-1 therapy management, where 4. Holmes, C. et al. 2014. Health systems implications of the 2013
inaccurate viral load results could lead to inappropriate therapy WHO consolidated antiretroviral guidelines and strategies for suc-
cessful implementation. AIDS 28(Suppl. 2):S231-S239.
or failing therapy.
5. Lee, F.J., J. Amin, and A. Carr. 2014. Efficacy of initial antiretroviral
These Six sigma statistical concepts have much broader implica- therapy for HIV-1 infection in adults: a systematic review and meta-
tions than those of a single patient management decision. From a analysis of 114 studies with up to 144 weeks’ follow-up. PLoS One
9:e97482.
therapy management perspective, virus with resistance-associated
mutations is more challenging to treat, with narrower windows for 6. Shaheen, M.A. and M. Idrees. 2015. Evidence-based consensus on the
diagnosis, prevention and management of hepatitis C virus disease.
therapy options. In addition, false-positive rates (blip frequency) World J. Hepatol. 7:616-627.
can add unnecessary costs due to extensive repeat testing or more 7. Webster, D.P., P. Klenerman, and G.M. Dusheiko. 2015. Hepatitis
expensive resistance testing, and increase anxiety for the patient. C. Lancet 385:1124-1135.
Tracking an assay and instrument performance with Sigma met- 8. Blick, K.E. 2013. Providing critical laboratory results on time, every
rics allows laboratories to identify defects in assays prior to down- time to help reduce emergency department length of stay: how our
stream patient management decisions and impact. laboratory achieved a Six Sigma level of performance. Am. J. Clin.
Pathol. 140:193-202.
Conclusions 9. Burnett, D. et al. 2010. Collective opinion paper on findings of the
2009 convocation of experts on quality control. Clin. Chem. Lab.
Six-Sigma analyses provide a useful tool to assay precision and Med. 48:41-52.
overall quality for both HIV and HCV load testing. The Sigma
10. Carlson, R.O., F. Amirahmadi, and J.S. Hernandez. 2012. A primer
metric translates abstract analytical performance characteristics on the cost of quality for improvement of laboratory and pathology
into tangible measures that can impact laboratory operations and specimen processes. Am. J. Clin. Pathol. 138:347-354.
patient outcomes. As diagnostic methods and antiviral treatments 11. Daley, A.T. 2006. Pro: lean six sigma revolutionizing health care of
have evolved, the demands for analytical performance of viral load tomorrow. Clin. Leadersh. Manag. Rev. 20:E2.
testing have increased. In previous decades, extreme precision may 12. Gras, J.M. and M. Philippe. 2007. Application of the Six Sigma
not have been as necessary, but today’s medical treatments demand concept in clinical laboratories: a review. Clin. Chem. Lab. Med.
45:789-796.
higher precision. Without setting a higher bar for analytical per-
13. Landek, D. 2006. Con: six sigma not always the right answer in the
formance, more blips and errors will occur in viral loads, and fewer clinical laboratory. Clin. Leadersh. Manag. Rev 20:E3.
desirable outcomes will be achieved. 14. Nanda, S.K. and L. Ray. 2013. Quantitative application of sigma
Sigma metrics can be implemented only with clinically defined metrics in medical biochemistry. J. Clin. Diagn. Res. 7:2689-2691.
TEa limits, which must relate directly to the treatment guidelines 15. Nevalainen, D. et al. 2000. Evaluating laboratory performance on
quality indicators with the six sigma scale. Arch. Pathol. Lab. Med.
for the disease. When these TEa limits are defined, a Sigma metric
124:516-519.
can be calculated and an objective judgment can be made about
16. Westgard, J.O. and S.A. Westgard. Assessing quality on the Sigma
method precision and acceptability. Once the Sigma metric is cal- scale from proficiency testing and external quality assessment surveys.
culated, that assessment can be leveraged into additional actionable Clin. Chem. Lab. Med. (In press.)
steps for the laboratory, for example, design of the appropri- 17. Westgard, S. 2013. Prioritizing risk analysis quality control plans
ate quality control procedures (i.e., the number of controls, the based on Sigma-metrics. Clin. Lab. Med. 33:41-53.
control limits, and, to some extent, the frequency of controls), as 18. Davis, C. et al. 2008. An international collaborative study to establish
well as estimations of the number of tests that will be required to a replacement World Health Organization International Standard for
human immunodeficiency virus 1 RNA nucleic acid assays. Vox Sang.
detect an error within the viral density range where medical deci- 95:218-225.
sions are to be made. In summary, the Sigma metric is not just a 19. Glaubitz, J. et al. 2011. Accuracy to 2nd International HIV-1 RNA
benchmark; it is the first step in a process of optimizing an assay’s WHO Standard: assessment of three generations of quantitative

Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier 109

 HIV-1 RNA nucleic acid amplification tests. J. Clin. Virol. 50:119- 25. Valdiserri, R.O. et al. 2013. Measuring what matters: development of
124. standard HIV core indicators across the U.S. Department of Health
20. Holmes, H. et al. 2011. Preparation and evaluation of the 1st inter- and Human Services. Public Health Rep. 128:354-359.
national standard for the quantitation of HIV-2 RNA in plasma. J. 26. Naeth, G. et al. 2013. Comparison of HIV-1 viral load assay perfor-
Virol. Methods 175:246-252. mance in immunological stable patients with low or undetectable
viremia. Med. Microbiol. Immunol. 202:67-75.
21. Saldanha, J. 1999. Standardization: a progress report. Biologicals
27:285-289. 27. Ruelle, J. et al. 2012. HIV-1 low-level viraemia assessed with 3 com-
mercial real-time PCR assays show high variability. BMC Infect. Dis.
22. Althoff, K.N. et al. 2014. Disparities in the quality of HIV care when 12:100.
using U.S. Department of Health and Human Services indicators.
28. Wiesmann, F. et al. 2014. Variation analysis of six HCV viral load
Clin. Infect. Dis. 58:1185-1189.
assays using low viremic HCV samples in the range of the clinical
23. Matthews, T.,et al. 2012. National performance measures within a decision points for HCV protease inhibitors. Med. Microbiol. Immu-
changing environment: how a federal agency developed and improved nol. (In press.)
the measurement for HIV care and treatment. J. Health Care Poor
29. Kessler et. al. 2015 Evaluation of the COBAS® AmpliPrep/COBAS®
Underserved 23:225-235. TaqMan® HCV Test, v2.0 and comparison to assays used in routine
24. Panneer, N. et al. 2014. HIV and hepatitis C virus infection in the clinical practice in an international multicenter clinical trial: The
United States: whom and how to test. Clin. Infect. Dis. 59:875-882. ExPECT study. J.Clin. Virol. 67-72.

110 Clinical Microbiology Newsletter 37:13,2015 | ©2015 Elsevier