IJMB

10.5005/jp-journals-10054-0044
Interpretation of External Quality Assurance: How to and How Not to
Original Article

Interpretation of External Quality Assurance:

How to and How Not to
1
Satish Ramanathan, 2Srinivas Chakravarthy, 3Smitha Menon, 4Thirumalai Nallathambi, 5Micheal Sunny

ABSTRACT One among the many potential errors in clinical labo-

Precision and accuracy are the two pillars of quality in analytical ratory is the analytical errors, which are bound to occur in
testing process of a clinical laboratory. External quality assur- the examination phase of testing process. The two most
ance (EQA) holds a major share in shaping the analytical quality significant errors in the analytical process are impreci-
from accuracy perspective. But this depends on how laboratory
sion and inaccuracy.2 The clinical laboratories worldwide
perceives an EQA, understands it, and uses it effectively for
inaccuracy assessment. External quality assurance has its own have developed two effective tools for the imprecision
advantages and limitations, including the commutability of EQA and inaccurate investigation; those being internal quality
sample, traceability of methods of comparison, the statistical control (IQC) tool for imprecision monitoring and EQA
procedure used for evaluation, etc. Our study discusses three
tool for inaccuracy assessment.3 In shaping the analytical
brief case reports through which we have tried to explore the
advantages and limitations of EQA. quality from accuracy perspective, EQA holds a major
share. But this depends on how a laboratory perceives an
Keywords: Clinical biochemistry, External quality assurance,
Quality control, Traceability, Trouble shooting. EQA, understands it, and uses it efficiently for inaccuracy
assessment. General perception in clinical laboratory
How to cite this article: Ramanathan S, Chakravarthy S, Menon S,
Nallathambi T, Sunny M. Interpretation of External Quality practice is that “an unacceptable EQA result means poor
Assurance: How to and How Not to. Indian J Med Biochem accuracy and otherwise.”4 But whether this perception
2018;22(1):1-5. holds true in all occasions and what are the factors to be
Source of support: Nil considered while interpreting an EQA forms the focus
of this study. Our case study includes three brief case
Conflict of interest: None
reports through which we have tried to understand how
to interpret and troubleshoot EQA.
INTRODUCTION
Clinical laboratory is a court of justice and laboratorians are MATERIALS AND METHODS
advocates of quality. Quality management system (QMS) We did this study at the Division of Clinical Biochemistry,
of a laboratory is its judiciary system. Laboratorians and Department of Laboratory Medicine, MIOT Hospitals,
advocates work in similar ways and move toward the same Chennai, Tamil Nadu, India. Our laboratory participated
goal, customer safety. Both of them make a continual stride in EQA program organized by the International Organi-
toward pinning in and eliminating out the endangered zation for Standardization (ISO) 17043:2010 accredited
threats to the customer safety. In a clinical laboratory, errors EQA provider (Bio-Rad). We included analysis of EQA
produce a constant threat to patients’ safety.1 These labora- results of three analytes, including serum immuno-
tory errors are omnipresent in all phases of total testing globulin G (IgG) (Vitros 5600, immunoturbidimetric),
process. Hence, we need a “round-a-clock” stringent vigi- creatinine kinase-MB (CK-MB) activity (Vitros 5600,
lant system, in the form of a QMS to identify, investigate, immunoturbidimetric inhibition), and copper (Vitros
and eliminate these errors and ensure patients’ safety. 5600, 3,5-Di-Br-PAESA 4-(3,5-dibromo-2-pyridylazo)-N-
ethyl-N-(3-sulphopropyl) aniline).
1
Head, 2Director, 3Quality Coordinator, 4Division Incharge
5 RESULTS AND DISCUSSION
Deputy Division Incharge
1
Department of Clinical Biochemistry, Madras Institute of Case 1
Orthopedics and Traumatology International, Chennai, Tamil Nadu
India On January 9, 2017, our laboratory encountered an analyt-
2-5
Department of Laboratory, Madras Institute of Orthopedics
ical threat in the form of serum IgG “outlier” in EQA (Bio-
and Traumatology International, Chennai, Tamil Nadu, India Rad serum proteins program). Serum IgG EQA sample
Corresponding Author: Satish Ramanathan, Head, Department was processed in Vitros 5600 (immunoturbidimetric).5
of Clinical Biochemistry, Madras Institute of Orthopedics and For serum IgG, EQA showed a Z score of +3.25, against
Traumatology International, Chennai, Tamil Nadu, India, Phone: the peer group comparison.6 Our laboratory conducted a
+919710467791, e-mail: satishraksha1980@gmail.com
detailed investigation to understand the IgG EQA threat.
Indian Journal of Medical Biochemistry, January-June 2018;22(1):1-5 1
Satish Ramanathan et al

We followed a structured approach to the investiga- of 20 data points. The lab mean was 687 mg/dL and
tion based on the “Flow Chart for handling deviating the lab SD was 37 mg/dL, while the manufacturer’s
EQA results” developed by External quality Control for mean and SD limits were 620 and 55 mg/dL. It was
Assays and Tests (ECAT) Foundation in the Netherlands7 ensured that this positive deviation of the laboratory
(Table 1). Six important aspects of investigation included mean from the manufacturer’s mean was not contrib-
evaluation of transcriptional error, presurvey issues, uted by an imprecise calibration of IgG.
sample receipt/handling errors, test performance errors, • With respect to measurand calibration, the laboratory
data handling (by the EQA provider) errors, and errors had performed calibration 10 days prior EQA sample
in interpretation of EQA result. At the end of the inves- processing as per manufacturer’s recommendations.5
tigation, we ruled out all possible causes of an “outlier” According to the manufacturer, the calibration showed
except the “test performance error.” The following were a “Passed” status and was deemed to be successful.
the observations made with respect to “test performance”: • Taking into consideration the previous EQA results
• Internal quality control during the period of EQA for IgG, no statistically significant “outliers” were
sample processing was within the control limits reported except for the current one. Careful observa-
established by the laboratory, though the Level 1 IQC, tion of the EQA results showed that an acceptable
which was in measurement range of EQA result (Bio- performance was displayed with previous EQA
Rad Immunology plus IQC), was constantly reported samples with values falling on the lower and middle
on higher side of the laboratory mean but within 2 range of the analytical measurement range (AMR),
standard deviation (SD) limits. The laboratory mean while the present “outlier” fell on the higher side of
and SD limits were established as per the policy AMR for serum IgG. Hence, a high index of diagnos-
adopted by the division, which included minimum tic suspicion of a “proportionate systematic error”

Table 1: Deviating EQA results

Types of errors Observation Case 1 Case 2 Case 3
Transcription error a Error in coding test results from the instrument NA NA NA
b Error in reporting test results to EQA organization
c Mixing up test results
d Report results with wrong units
e Report the wrong method and/or equipment
f Error in data entry by EQA provider
Presurvey issues a The EQA provider distributed by accident an inappropriate sample NA NA NA
b Error in sample labeling
c Error in packaging the samples
d Error in distribution of samples
e Problem with sample stability
g Problem with sample homogeneity
h Error in the instruction letter of EQA provider
Sample receipt/ a Problems with the receipt of samples NA NA NA
handling b Inappropriate storage of samples till use
c Problems with reconstitution of samples
d The instructions were not followed properly by the participant
Test performance a Change in the instructions of manufacturer NA NA NA
b Was there a problem with the equipment
c Was there a problem with the reagents
d Was there a problem with the IQC samples
e Was there a problem with the test performance Applicable
Data handling EQA a The statistical procedure used is probably not appropriate for the NA NA NA
provider distribution of the test
b Error in establishment of the AV Applicable Applicable
c Error in presentation of results NA NA
Report and a Deviation in accordance with previous EQA results NA NA NA
interpretation b Large variation in EQA results for the method used
c Deviation is systematic for all EQA samples
d Repeated analysis showed similar deviation
e Source for the deviation is unknown
NA: Not applicable

2
 IJMB

Interpretation of External Quality Assurance: How to and How Not to

was considered.8 One of the recommended quality Y-intercept should be 0. Y-intercept is considered as
practices to diagnose this error included calibration an indicator of constant systematic error that affects
verification.9 But till then, our laboratory had not the comparability of results constantly across the
adopted a practice of verification of calibration expect measuring interval.
for IQC check after measurand calibration. The observed slope was 1.109. The observed slope
• Calibration verification was performed using IgG was compared against the ideal slope (1.0). The criteria
calibrators as testing materials obtained from Ortho for acceptable performance were established as:
Clinical Diagnostics.9 These were a new set of similar Ideal slope ± TEa/100
calibrators that had been used for IgG calibration. Five Acceptable performance = 1.0 ± (8/100) = 0.92
calibrators of concentrations spanning the AMR were to 1.08.
selected as testing materials. The samples were run in • The observed slope showed a statistically significant
duplicates and the average of the observed values was positive deviation from the ideal slope suggesting a
compared against the assigned values (AVs) provided proportionate systematic error.8 Hence, we performed
by the manufacturer by using: a fresh calibration followed by calibration verification
• Difference plot to ensure that this error was eliminated.
A visual assessment of the data was done by using This case showed how an EQA can guide us toward
a difference plot which was created by comparing the continual improvement of quality practice in the form
observed difference (difference between the AV and of calibration verification.
the observed value) against the AV. The observed
difference (%bias) seemed to be significantly high. Case 2
This was followed by a statistical assessment of the It was March 20, 2017, when EQA picked up a statistically
difference plot which was done by comparing the significant “outlier” of serum CK-MB activity. External
observed bias against the allowable bias as per desir- quality assurance (Bio-Rad Cardiac markers) reported
able biological variation (BV) specifications.9,10 The CK-MB activity “outlier” in two equipments (Vitros
observed %bias (8.79%) was greater than the allowable 5600-1 and Vitros 5600-2) by CK-MB immunoinhibition
bias (4.3%) (Graph 1A). method11 with a Z-score of –3.85 and –3.92 in both the
• Linear regression plot equipments.
A linear regression graph was plotted to find out Our laboratory did a structured root cause analysis
the slope (Graph 1B), wherein slope describes the for these outliers (Table 1). Based on the findings of
angle of the line that provides the best fit to the test the investigation, we zeroed in the possibility of a data
and the comparative results, the ideal slope being 1.0. handling error by the EQA provider.7 The laboratory
Slope is considered as an indicator for proportionate advocated possibilities of three types of errors with
systematic error, wherein the magnitude of error respect to handling of data by the EQA provider, which
increases as the concentrations get higher. included:
Y-intercept describes the point where the line • Inappropriate statistical procedure used by EQA
of best fit intersects with the Y-axis. Ideally, the provider for evaluation of dataset.

A B
Graphs 1A and B: (A) Difference plot—IgG; (B) linear regression plot—IgG

Indian Journal of Medical Biochemistry, January-June 2018;22(1):1-5 3

Satish Ramanathan et al

• Error in establishment of “AV” • Calibration verification: Three CK-MB calibrators

• Error in presentation of results by EQA provider. acquired from Ortho Clinical Diagnostics were used
Among these three errors, error (a) was ruled out, as testing materials.9 The samples were run in dupli-
since the EQA whom we had enrolled with was an ISO cates and the average of the observed values was
17043:2010 accredited provider and hence, the statistical compared against the assigned values provided by
procedures used by EQA provider were in compliance the manufacturer by using a difference plot, wherein
with international standard. We rule out error (c), which the observed bias (0.9%) was compared against the
involved wrong presentation of results by EQA provider allowable bias (7.8%) as per desirable BV specifica-
(when the EQA provider wrongly links the laboratory tions.9,10 The comparison yielded acceptable results
results for a specific method to another method). After (Graph 2A). This was followed by slope estimation
having ruled out error (a) and (c) as possibilities, we by linear regression plot (Graph 2B). The observed
zeroed in error (b) as a potential cause behind the outlier. slope (0.99) was compared against the ideal slope as
This was based on the evidence that our laboratory results per the criteria for acceptable performance (Ideal slope
(from two equipments) were compared against the “AV” ± TEa/100, where TEa = 24.1%). The slope obtained
established by EQA in consensus with a group of par- from our study was within the acceptable limits
ticipating laboratories not specific for our methodology. (0.76 to 1.24) and hence, the analytical performance
The gross discrepancy evident in CK-MB values when of CK-MB was considered acceptable.
compared across methodologies by the EQA provider is This case illustrated a case scenario wherein EQA
explained by lack of metrological traceability of calibra-
had falsely branded CK-MB activity as an “outlier”
tors across different methods of CK-MB measurement.11
which was proved otherwise through detailed inves-
This was the fourth instance in previous 6 months of EQA
tigations by our laboratory.
cycle, wherein CK-MB was branded outlier, based on
comparison with AV not specific for our method though. Case 3
Assays are not standardized for measurands for which
the calibrators are not traceable to a reference method On May 31, 2017, the EQA provider (Bio-Rad) released the
or a reference material.12 Hence, the laboratory has to reports of analysis of clinical chemistry (monthly) program
evaluate the EQA results of such measurands against the for measurands with serum-based matrix, which included
method-specific consensus AV, if provided by the EQA serum copper (Vitros 5600, 3,5-Di-Br-PAESA 4-(3,5-dibromo-
and not the total group AV. 2-pyridylazo)-N-ethyl-N-(3-sulphopropyl) aniline).13 All
The EQA having declared CK-MB as an “outlier,” but measurands were reported to have acceptable performance
by comparing against a method nonspecific AV, we set out according to the EQA provider including serum copper
to gather evidence against the EQA to prove innocence (Z-score: 1.63). Z-score < 2 is considered as an acceptable
of CK-MB activity including: result according to ISO/IEC standard 17043:2010.6 But our
• In our laboratory, CK-MB EQA results were compared laboratory has adopted a quality practice of reviewing all
between two equipments (bias% is 0.29%). The com- EQA reports. The focus of our investigation turned toward
parison being made based on allowable %bias (7.1%) serum copper EQA performance. Though, as per the EQA
as per desirable BV specifications.10 provider, copper showed an acceptable performance, we

A B
Graphs 2A and B: (A) Difference plot—CKMB; (B) linear regression plot—CKMB

4
 IJMB

Interpretation of External Quality Assurance: How to and How Not to

Table 2: Serum copper EQA result

Comparison n Mean Our result SD CV% U Z-score Bias%
Mode-based comparison* 52 63.9 µg/dL 95 µg/dL** 19.1 µg/dL 29.9% 6.63 1.63 48.7%
*Comparison against the total group AV, which is not specific to our methodology; **Vitros 5600, 3,5-Di-Br-PAESA 4-(3,5-dibromo-
2-pyridylazo)-N-ethyl-N-(3-sulphopropyl) aniline; CV: Coefficient of variation

were not satisfied with the comparison, since we observed in comparison not against a reference method but with
a gross deviation of our value (95 µg/dL) from the AV of other participating laboratories. Hence, it is the need of
EQA (63.9 µg/dL) (Table 2). the hour for the laboratory medicine specialists to under-
The observed %bias between our value and AV of stand the pros and cons of EQA and learn to interpret and
EQA (48.7%) was significantly greater than the allowable troubleshoot EQA.
%bias (3.6%) as per desirable BV specifications.10 But in
contradiction, EQA showed an acceptable performance REFERENCES
in terms of Z-score, against the total group AV, which was 1. Ottomano C. Errors in medicine and errors in laboratory medi-
not specific to our methodology. Copper (in Vitros 5600) cine: what is the difference? .Blood Transfus 2010 Apr;8(2):
being a measurand without metrological traceability is 79-81.
bound to have an unreliable comparison across method- 2. Badrick T. Quality leadership and quality control. Clin
ologies12 unless compared against a higher order method Biochem Rev 2003 Aug;24(3):81-93.
like atomic absorption spectroscopy (AAS).14 Though 3. Coucke W, Soumali MR. Demystifying EQA statistics and
reports. Biochem Med 2017 Feb;27(1):37-48.
such a comparison was not available with the EQA
4. Salle B, Meijer P, Thomas A, Simundic AM. Special issue
provider, peer group analysis of labs performing copper
on external quality assessment in laboratory medicine-
with AAS was available for review by the EQA provider. current challenges and future trends. Biochemia Med 2017
Hence, we tried to compare our EQA result against Feb;27(1):19-22.
mean of EQA of labs performing AAS. The observed 5. Immunoglobulin G. Vitros chemistry products IgG reagent.
bias% was 2.8%, which was within the desirable bias% Pub No. J21375_EN, Version 6.0; 1-10.
(3.6 %) according to BV desirable specifications. This com- 6. International Organization for Standardization/International
parison is done based on the laboratory’s quality policy Electrotechnical Commission. Conformity assessment—
general requirements for proficiency testing. ISO 17043.
traceable to Clinical & Laboratory Standards Institute
Geneva: ISO/IEC; 2010.
(CLSI) QMS 24-ED3:2016—“Using proficiency testing 7. Gunn B, Berge K, Piet M. Interpretation of EQA results and
and alternative assessment to improve medical labora- EQA-based trouble shooting. Biochem Med 2017 Feb;27(1):
tory quality-3rd edition.”15 As per CLSI, when a clinical 49-62.
laboratory encounters a scenario, wherein it finds itself 8. Westgard JO. Basic method validation, 3rd edition.The pub-
without a genuinely comparable peer group in EQA, lisher is Westgard Corporation Publisher Location: Madison,
and the EQA provider too is unable to grade its results, Wisconsin 62-64.
9. Available from: https://www.westgard.com/cal-verification-
evaluation of these results must be done by the laboratory
criteria.htm. Basic method validation. Calibration verification:
itself through comparison with an appropriate designate Defining criteria for acceptable performance.
method in the EQA provider’s summary report and in 10. Available from: https://www.westgard.com/biodatabase1.
that case, the laboratory can use that method’s mean and htm. Quality requirements. Desirable Biological Variation
SD to do its own evaluation. database Specifications.
This investigation helped us to ascertain the quality 11. CKMB. Vitros Chemistry Products CKMB Slides. Pub.No.
of performance of serum copper in our lab. This case MP2-48_EN, Version 8.0, 1-13.
12. Koumantakis G. Traceability of measurement results. Clin
exposed the limitations of EQA which produced a con-
Biochem Rev 2008 Aug;29(Suppl 1):S61-S66.
tradictory report (with an acceptable Z-score, but an
13. Copper (Cu) Manual. RANDOX, revised 2nd October 07 ds.
unacceptable %bias) and kept the laboratory guessing 14. Beckett JM, Hartley TF, Ball MJ. Evaluation of Randox
about the quality of its performance. colorimetric serum copper and zinc assays against atomic
absorption spectroscopy. Ann Clin Biochem 2009 Jul;46(Pt 4):
CONCLUSION 322-326.
15. Clinical and Laboratory Standards Institute. Using profi-
External quality assurance is an effective quality tool ciency testing and alternative assessment to improve medical
for accuracy assessment but with its own limitations, laboratory quality. 3rd edition. Wayne, PA: CLSI; 2016. CLSI
especially when it judges a laboratory’s performance document QMS 24-ED3.

Indian Journal of Medical Biochemistry, January-June 2018;22(1):1-5 5