Journal of Biomedical Informatics 51 (2014) 80–85

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier.com/locate/yjbin

The use of fault reporting of medical equipment to identify latent design

flaws
C.J. Flewwelling a,b,⇑, A.C. Easty a,c, K.J. Vicente d, J.A. Cafazzo a,b,e
a
Institute of Biomaterials and Biomedical Engineering, University of Toronto, Rosebrugh Building, RM 407, 164 College Street, Toronto, ON M5S 3G9, Canada
b
Healthcare Human Factors, University Health Network, 4th Floor R. Fraser Elliot Building, 190 Elizabeth Street, Toronto, ON M5G 2C4, Canada
c
HumanEra, University Health Network, 4th Floor R. Fraser Elliot Building, 190 Elizabeth Street, Toronto, ON M5G 2C4, Canada
d
Department of Mechanical and Industrial Engineering, University of Toronto, 5 King’s College Road, Toronto, ON M5S 3G8, Canada
e
Institute of Health Policy, Management and Evaluation, University of Toronto, Health Sciences Building, 155 College Street, Suite 425, Toronto, ON M5T 3M6, Canada

a r t i c l e i n f o a b s t r a c t

Article history: Background and purpose: Poor device design that fails to adequately account for user needs, cognition,
Received 2 December 2013 and behavior is often responsible for use errors resulting in adverse events. This poor device design is also
Accepted 7 April 2014 often latent, and could be responsible for ‘‘No Fault Found’’ (NFF) reporting, in which medical devices sent
Available online 16 April 2014
for repair by clinical users are found to be operating as intended. Unresolved NFF reports may contribute
to incident under reporting, clinical user frustration, and biomedical engineering technologist inefficacy.
Keywords: This study uses human factors engineering methods to investigate the relationship between NFF report-
No Fault Found
ing frequency and device usability.
No problem found
No trouble found
Material and methods: An analysis of medical equipment maintenance data was conducted to identify
Clinical engineering devices with a high NFF reporting frequency. Subsequently, semi-structured interviews and heuristic
Human factors evaluations were performed in order to identify potential usability issues. Finally, usability testing was
Medical device design conducted in order to validate that latent usability related design faults result in a higher frequency of
NFF reporting.
Results: The analysis of medical equipment maintenance data identified six devices with a high NFF
reporting frequency. Semi-structured interviews, heuristic evaluations and usability testing revealed that
usability issues caused a significant portion of the NFF reports. Other factors suspected to contribute to
increased NFF reporting include accessory issues, intermittent faults and environmental issues. Usability
testing conducted on three of the devices revealed 23 latent usability related design faults.
Conclusions: These findings demonstrate that latent usability related design faults manifest themselves
as an increase in NFF reporting and that devices containing usability related design faults can be identi-
fied through an analysis of medical equipment maintenance data.
Ó 2014 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

1. Introduction frequent cause of patient harm [3]. In some cases medical devices
have poorly designed and difficult to use human system interfaces
Evidence suggests that adverse events associated with medical [4]. As a result, there is an increasing interest in incorporating
devices are more often the result of use error than device human factors engineering (HFE) principles in the design and eval-
malfunction [1]. One study found that 82% of all preventable errors uation of medical devices. However, the use of HFE principles
involving anaesthesia devices were due to use error [2], while a within healthcare is still not widespread [5]. There remain
study of infusion pump errors found that use errors were the most instances in which usability related design flaws are identified by
users as a device malfunction. When investigated, conclusions of
Abbreviations: NFF, No Fault Found; UHN, University Health Network; WRHA,
‘‘No Fault Found’’ or ‘‘cannot replicate problem’’ are often reached.
Winnipeg Regional Health Authority; NSAHO, Nova Scotia Association of Health These outcomes cannot be used to mitigate the situation and may
Organizations; VIHA, Vancouver Island Health Authority; BMET, biomedical engi- contribute to user frustration and incident under-reporting. In this
neering technologist; HFE, human factors engineering. study, we sought to identify devices that contain latent usability
⇑ Corresponding author at: Healthcare Human Factors, University Health Net-
related design flaws using medical equipment maintenance data
work, RFE 4th floor, 190 Elizabeth Street, Toronto, ON M5G 2C4, Canada.
E-mail address: christopher.flewwelling@uhn.ca (C.J. Flewwelling).
and human factors techniques.

http://dx.doi.org/10.1016/j.jbi.2014.04.009
1532-0464/Ó 2014 The Authors. Published by Elsevier Inc.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).
 C.J. Flewwelling et al. / Journal of Biomedical Informatics 51 (2014) 80–85 81

2. Background devices with the greatest probability of containing latent usability

related design flaws the NFF reporting rate was normalized in an
Medical devices that are suspected by clinical users to be defec- attempt to account for frequency of device use. There are no hospi-
tive are typically sent to a hospital’s biomedical engineering service tal databases that outline the frequency of device use. As such,
department for troubleshooting and repair by biomedical engineer- equipment inventory count was used in normalizing the reporting
ing technologists and technicians (BMETs). Maintenance reports rate as there is presumed to be a correlation between frequency of
are routinely completed before returning the device back into ser- device use and equipment inventory levels.
vice. The maintenance reporting data allow BMETs to categorize
the device failure mode(s) and specify what repair actions were 3.2. BMET semi-structured interviews
taken. One of the failure mode categories BMETs have the option
of selecting is ‘‘No Fault Found’’ (NFF). The NFF category is used In order to contextualize and qualitatively confirm the results of
when a device sent for repair is found to be operating as intended. the database analysis a series of semi-structured interviews with
Usually in these instances the device is placed back into service BMETs were conducted. Eight BMETs from UHN were recruited
without detailed investigation into the root cause [6]. Explanations to participate in semi-structured interviews, which lasted between
for users believing devices to be defective include the failure of 30 and 60 min each. Participants work experience ranged from 5 to
device designs to adequately account for user needs, cognition, 25 years. BMETs were not made aware of the results of the data-
and behavior, environmental factors, and intermittent faults. base analysis. The interviews were recorded and transcribed by
Draper [7] provides a case study in which a usability related the investigator. The transcripts from each session were then thor-
design flaw in a medical device was uncovered by an investigation oughly reviewed and initial specific low level codes were assigned
initiated as a result of recurrent NFF reporting. An incident to each word, phrase, or paragraph of text associated with an indi-
occurred at a hospital in which a syringe pump delivered a drug vidual idea. The coded transcripts were then iteratively reread and
dose in half the anticipated time. When the nursing staff could reassessed allowing for codes to be combined, divided, added or
not reproduce the error, the pump was sent to Biomedical Engi- removed. Inferences were then drawn about what the various
neering where no problem was found. After recalling several sim- codes might represent. Using these inferences, similar codes were
ilar NFF reports, BMETs subsequently performed a more detailed grouped in order to form broad overarching thematic statements.
analysis. The analysis revealed that the user may have overridden Finally, each theme was compared to the original dataset in order
the standard syringe size, instead programming the pump to to assess whether or not it appropriately described the data.
expect a syringe size numerically equivalent to the dosage rate. It
was demonstrated that this change results in an over infusion. 3.3. Heuristic evaluation
The technologist concluded that this error could occur either
through incorrect understanding of the difference between the Next, heuristic evaluations were conducted in order to deter-
dosage and syringe size or by accidentally pressing the up/down mine the likelihood that the devices identified as a result of the
arrows on the pump thereby inadvertently changing the syringe database analysis contain latent usability related design faults.
size. While the design flaw was uncovered as a result of recurrent The heuristic evaluations involved applying a set of usability prin-
NFF reporting, the investigation was only initiated as a result of a ciples to systematically evaluate each device. Two investigators,
technologist remembering previous incidents. acting independently, evaluated each device, recording any heuris-
In this study we aimed to build on the premise of the case study tics violations. The usability heuristics used were: consistency and
by utilizing medical equipment maintenance data in order to pro- standards, visibility of system state, match between system and
actively identify devices which may contain latent usability related world, minimalism, minimizing memory load, informative feed-
design flaws that could lead to adverse events. back, flexibility and efficiency, good error messages, prevention
of errors, clear closure, reversible actions, use of users’ language,
users are in control, and help and documentation [8]. After each
3. Methods and materials device was evaluated, the lists of heuristics violations were com-
bined and each investigator independently assessed the severity
3.1. Database analysis of the identified violations.

In order to identify the devices most frequently associated with 3.4. Usability testing
NFF reporting, a quantitative analysis of medical equipment
maintenance data from University Health Network (UHN, Toronto, Three devices (a defibrillator, a feeding pump, and a thermom-
Canada), Nova Scotia Association of Health Organizations (NSAHO), eter) were selected for usability testing based on the findings from
Winnipeg Regional Health Authority (WRHA), and Vancouver the database analysis, BMET interviews, and heuristic evaluations.
Island Health Authority (VIHA) was conducted. UHN, NSAHO, The testing involved clinical users performing representative tasks
WRHA and VIHA own approximately 2500, 3500, 7000 and 4000 with the devices in a simulated environment in order to reveal
equipment models respectively. Medical equipment maintenance usability design flaws. The usability testing approach is strongly
data from 2003 to 2011 was analyzed in order to identify all rooted in theories and methods from the field of cognitive science
reports of unscheduled equipment repair in which the BMET [9] and is recognized by the FDA as a method that should be used
responsible for repair could find no fault. NFF reports were identi- in identifying potential use related hazards [10]. For the purpose of
fied based on data contained within the failure mode field and free this investigation, usability tests were conducted in order to assess
text comment field. Reports that contained ‘‘problem not found’’, whether the proposed methodology was successful in identifying
‘‘No Fault Found’’ (or variations thereof) as a fail mode were devices with latent usability related design flaws. Eleven registered
selected. Reports that contained free text comments indicating that nurses and ten medical doctors from UHN participated in the eval-
no fault was found were also selected. Each NFF report was then uation of the defibrillator. Ten registered nurses from UHN partic-
associated with a particular equipment model so that the number ipated in the evaluation of feeding pump and thermometer. Testing
of NFF reports per model could be determined. was conducted in a high fidelity simulation lab set up to resemble
The more frequently a device is used the greater the probability an intensive care unit. The environment contained three manne-
that it will be sent for repair. Therefore, in order to identify the quins to simulate patients. Hospital sound effects were played
82 C.J. Flewwelling et al. / Journal of Biomedical Informatics 51 (2014) 80–85

through overhead speakers during the testing sessions in order to 5. Use errors (differences between models of similar devices).
simulate ambient hospital noise. A facilitator remained in the lab 6. Faults with a device accessory (e.g., faulty blood pressure
for the duration of the evaluation acting as the participant’s col- cuff).
league and providing the participant with details regarding the 7. Device availability (it is suspected users spend more time trou-
testing scenarios. Video cameras were used to capture participants bleshooting devices for which a replacement is not readily
interacting with the devices. Additionally, a human factors special- available).
ist observed each session from behind a one-way mirror and doc- 8. Intermittent faults.
umented details of the participants’ interaction with the devices. 9. Complacency (users assuming device is broken because it has a
Scenarios included tasks frequently associated with use of each bad reputation of breaking down).
particular device as well as tasks that were suspected to be associ-
ated with NFF reporting. As the devices investigated are used by BMETs were also able to provide context on a device specific
the testing participants on a regular basis no opportunity for addi- basis. For example, it was suggested that inadequately designed
tional training was provided. pacing and cardioversion synchronization features were
responsible for NFF reporting associated with the Heartstart XL
4. Results Defibrillator.

4.1. Database analysis results

4.3. Heuristic evaluations results
Analysis of medical equipment maintenance data resulted in
the identification of the ForceFx Electrosurgical Machine (Valley- Heuristic evaluations of the selected devices by the two
lab, Boulder, Colorado), Suretemp 678 Thermometer (Welch Allyn, investigators revealed a total of 112 usability issues. The two
Skaneatele Falls, New York), 52000 Vital Signs Monitor (Welch investigators had an overall inter-rater reliability (weighted
Allyn, Skaneatele Falls, New York), Heartstart XL Defibrillator (Phi- Cohen’s Kappa) of 0.77 indicating substantial agreement. Fig. 1
lips, Andover, Massachusetts), Kangaroo Enteral Feeding Pump provides an overview of the violations by device and severity.
(Covidien, Dublin, Ireland) and 245 Blood Warmer (Arizant, St. Violations identified during the heuristic evaluations of the
Paul, Minnesota) as devices associated with a high NFF reporting ForceFx Electrosurgical Machine, 678 Suretemp Thermometer,
rate. As illustrated in Table 1, NFF reporting rates associated with 52000 Vital Signs Monitor, Heartstart XL Defibrillator, and
the identified devices exceed organizational averages. This indi- Kangaroo Feeding Pump correlated well with the results of the
cates that there is an increased potential that the devices contain database analysis and semi-structured interviews. For instance,
latent design flaws. database analysis and interviews with BMETs revealed that the
pacing, synchronized cardioversion, and ECG display functions
4.2. Semi-structured interview results are features of the Heartstart XL Defibrillator often associated with
NFF reporting. The heuristic evaluation revealed a number of
Discussions with BMETS qualitatively confirmed the results of
potential usability issues associated with these features including
the database analysis. BMETs associated eleven devices with fre-
low default ECG wave amplitude, lack of autogain, failure to notify
quent NFF reporting. Of the eleven devices, eight were among
the user that they must depress the shock button for an entire
the twenty devices most frequently associated with NFF reporting
cycle to cardiovert, and failure to notify the user of potentially
at UHN. Furthermore, of the six devices selected during the data-
inverted leads.
base analysis for further investigation, only the ForceFx Electrosur-
gical Machine was not identified by BMETs. BMETs were also able Conversely, violations identified during the heuristic evaluation
to provide context as to why NFF reporting may have occurred. of the 245 Blood Warmer do not correspond well with the results
Thematic analysis of the BMET interview data indicated nine per- of the database analysis and semi-structured interviews. Violations
ceived causes: are primarily related to the lack of alert when a bag is incorrectly
inserted or empty. Use errors associated with these violations
1. Use errors (unspecified). would likely become apparent to a user prior to the device being
2. Use errors (battery needs charging or incorrectly inserted). sent for repair. As such, it is more likely that intermittent faults
3. Use errors (new staff unfamiliar with device). or environmental factors are the cause of the NFF reporting associ-
4. Use errors (environmental). ated with the 245 Blood Warmer.

Table 1
NFF reporting rates associated with select devices.

Device UHN NSAHO WRHA VIHA

Number of NFF Normalized Number of NFF Normalized Number of NFF Normalized Number of NFF Normalized
reports NFF rate reports NFF rate reports NFF rate reports NFF rate
ForceFx Electrosurgical 12 1.2 19 0.73 28 0.41 28 1.22
Unit
678 Suretemp 329 1.79 6 0.43 155 0.48 7 1.17
Thermometer
52000 Vital Signs 11 0.92 80 1.63 64 0.66 26 1.08
Monitor
Heartstart XL 64 2.29 27 1.93 Device not used Device not used
Defibrillator
Kangaroo Enteral 192 2.7 Device not used 28 0.11 51 0.28
feeding pump
245 Blood Warmer 121 3.78 Device not used 0 0.00 1 0.01
Site averages NA 0.56 NA 0.43 NA 0.28 NA 0.26
 C.J. Flewwelling et al. / Journal of Biomedical Informatics 51 (2014) 80–85 83

1. Ensure human factors principles are considered within the equip-

ment procurement process.

Healthcare organizations should consider augmenting their

procurement processes with a comparative usability study in order
to evaluate the safety and ease of use of devices they are consider-
ing purchasing [11]. By assessing how a new device or system
might impact users and the use environment, organizations may
be able to proactively implement mitigation strategies and develop
training that effectively addresses known usability issues. Procur-
ing devices that are intuitive to use and that are easily integrated
into the current workflow of users should reduce device use errors
and, by extension, NFF reporting frequency.

2. Analyze medical equipment maintenance data for devices associ-

ated with a high NFF reporting rate.

Ideally, every NFF report would be fully investigated until the

cause, whether usability related or otherwise, was determined.
Such an approach would however require significant resources.
Fig. 1. Summary of heuristic violations. Note that having the greatest number of The results of this investigation indicate that database analysis
total violations does not necessarily mean that a given product was the most provides an effective means of identifying NFF trends that may
problematic since the distribution of violations at each severity level is not
be of interest. Therefore, in order to focus resources, it is recom-
consistent. Furthermore, severe heuristic violations may not necessarily translate to
a severe patient outcome. mended that organizations analyze medical equipment mainte-
nance data in order to identify devices for detailed investigation.
While this step is essential to the proposed methodology, organiza-
4.4. Usability testing results
tions are able to account for available resources by adjusting how
often data is analyzed and how many of the identified devices are
Usability testing of the Heartstart XL Defibrillator, Kangaroo
fully investigated.
Feeding Pump, and Suretemp 678 Thermometer revealed a total
of 23 usability related design flaws. Tables 2–4 outline the design
3. Validate the results of the database analysis with BMETs.
flaws identified for each device respectively.
The results indicate that BMETs are capable of providing valu-
5. Discussion able context to the database analysis. BMET discussions were dem-
onstrated to be a fundamental part of the process of identifying
Usability testing of the Heartstart XL Defibrillator, Kangaroo why NFF reporting occurred. Therefore it is strongly recommended
Feeding Pump, and Suretemp 678 Thermometer revealed a total that BMETs participate in the process of selecting devices for fur-
of 23 design flaws. The usability testing results therefore confirm ther investigation.
that medical equipment maintenance data can be used to identify
devices that contain latent usability related design flaws. Further- 4. Conduct heuristic evaluations of devices associated with a high NFF
more, the semi-structured interviews and heuristic analyses rate in order to determine the most likely cause of the reporting.
results correlated closely with the usability testing results indicat-
ing that they provide an effective means of characterizing the The database analysis and semi-structured interviews were
possible nature of the design flaws. Based on the results of the shown to identify devices, such as the Heartstart Defibrillator
investigation, the following methodology is recommended both for and Kangaroo Pump, which contained usability related design
use in reducing the frequency of NFF reporting and for use in iden- faults. However, the database analysis and interviews also identi-
tifying devices that contain latent usability related design flaws. fied devices, such as the 245 blood warmer, which did not contain

Table 2
Heartstart XL Defibrillator design flaws.

Design flaw ID Design flaw Task(s) affected

D1 Location of accessory connection Ensure pads/paddles are connected
D2 Inadequate pads disconnect error message Ensure pads/paddles are connected
D3 Lack of error message explaining failed SYNC Ensure sync mode disengaged when applicable
Ensure QRS gain sufficient
D4 Lack of error message explaining shock button inoperable Locate and press shock button
D5 Lack of lead placement alert Fix inverted waveform
D6 Inadequate lead placement diagram Fix inverted waveform
D7 Lack of a system state indicator for SYNC disengaged Ensure sync mode engaged
D8 QRS gain adjustment is not automatic Ensure QRS gain is sufficient
D9 Device does not default to lead II Ensure QRS gain is sufficient
D10 No alert indicating to depress shock button until shock has been delivered Hold shock button for duration of cardioversion
D11 Inadequate pacer key inactive message Enter pacing mode
D12 Location and labeling of pacing controls Set pacing current
Start pacing
D13 Inadequate system state message indicating pacing mode Ensure fixed pacing
D14 Pacing status message is unclear Recognize pacing has been stopped and resolves
84 C.J. Flewwelling et al. / Journal of Biomedical Informatics 51 (2014) 80–85

Table 3 features. For example, users had difficulty using the VTBD mode
Kangaroo Enteral feeding pump design flaws. on the pump and axillary mode on the thermometer as they were
Design Design flaw Task(s) affected more familiar with the rate and oral modes respectively. As such, it
flaw ID should be ensured that in service training covers all features users
P1 Method of entering biotech Turn on pump without entering experience difficulty with.
mode biotech mode
P2 Lack of units and labels on Program pump in VTBD mode 7. Consider redirecting training resources in order to retrain users on
display screen
Silence and acknowledge alarms
the identified devices.

P3 Complicated button Switch from VTBD mode to Rate

sequences mode mid cycle
Higher NFF rates might reflect that a device is less intuitive to
Switch from VTBD mode to Rate use. It may therefore be desirable to redirect limited training
mode end of cycle resources away from devices users are able to operate easily and
P4 Ambiguous button labels Switch from VTBD mode to Rate effectively and instead focus training on devices that are more
mode mid cycle problematic. Healthcare organizations should however consider
Switch from VTBD mode to Rate the risks associated with use of a given device prior to redirecting
mode end of cycle
any resource in order to ensure that regular and adequate training
Silence and acknowledge alarms
is provided for higher risk devices.
P5 Lack of system state Switch from VTBD mode to Rate
messages mode mid cycle
Switch from VTBD mode to Rate 8. Provide feedback to the manufacturer in order to inform the design
mode end of cycle of future versions of the devices.
P6 Unclear alarm messages Resolves feed empty alarm
Resolves occlusion alarm User performance and behavior modifications that can be dri-
Recognize VTBD Alarm and ven by increased training or modified policy may provide a basic
program new cycle mitigation to any identified usability related design flaws. Such
P7 Volume clear indicator Programs pump in rate mode modifications are however not guaranteed to be effective as staff
Programs pump in VTBD mode may not attend or be fully engaged during training. Furthermore,
the impact of even the most effective training will decay over time.
Therefore, it is preferable to address the root cause of any design
Table 4 flaws through system based design modifications. Manufacturers
678 Suretemp Thermometer design flaws.
are often open to feedback from healthcare organizations. As such,
Design flaw ID Design flaw Task(s) affected it is highly recommended that healthcare organizations inform
T1 Lack of calibration indicator Wait for calibration manufacturers of any usability related design flaws they uncover.
Take temperature orally
Take axillary temperature 5.1. Implementing the proposed methodology
T2 Mode toggle button Switch to axillary mode
Healthcare organizations should consider utilizing the proposed
methodology on a regularly scheduled basis. The scheduling fre-
usability related design faults. As such, heuristic evaluations quency and number of devices investigated in each instance should
should be performed on all devices associated with a high NFF rate be based on each individual organization’s resource availability.
in order to determine if usability related design flaws are in fact the While it is preferable to employ all of the outlined steps,
most likely cause of the frequent NFF reporting. Heuristic analysis resources may not always be available. Incorporating Step 1
is a technique that is relatively easy to learn and use proficiently. (human factors informed procurement) is highly recommended
Therefore organizations without dedicated human factors staff as it serves to proactively address many of the concerns that are
should still be able to effectively assess whether specific NFF reports otherwise only addressed in a reactive manner. A decision not to
may be associated with latent usability related design flaws. include Step 1 will not however impede incorporating the remain-
ing steps of the methodology. Steps 2–4 (database analysis, BMET
5. If resources are available, usability testing should be conducted in discussions, heuristic analyses) form the basis of the methodology,
order to validate results. and as such it is expected that removing any of these steps would
significantly impact the efficacy. Step 5 (usability testing) is a con-
Usability testing was a fundamental part of conclusively identi- firmatory step. Though it is desirable to conduct usability testing if
fying the presence of latent usability related design faults in a resources permit, it is nevertheless believed that a sufficient
number of the devices investigated. However, it is time consuming, understanding of the cause of NFF reports can still be obtained
and requires a testing environment that may not be available. Fur- even if Step 5 is omitted. While it is not necessary to incorporate
thermore, usability testing results correlated well with the semi- all of Steps 6 through 8 (targeted training, redirected training,
structured interview and heuristic analyses results. Therefore, informing manufacturer), in order to benefit from the methodology
while it is desirable to conduct usability testing, it is nevertheless mitigating actions must be taken. Organizations should consider
believed that a sufficient understanding of the cause of the NFF the preferred form of mitigation on a case-by-case basis. Steps 2
reporting can still be obtained even if it is not feasible to conduct and 3 can be undertaken simultaneously as can Steps 6 though 8.
usability testing. All the other steps should be undertaken sequentially.
If the most likely cause is usability related design flaws: In order to assess whether the methodology improves BMET
efficiency and reduces clinical user confusion, organizations may
6. Ensure that device training covers the features associated with the want to track whether its implementation has resulted in a reduc-
NFF reporting (even if they are not the most commonly used features). tion in NFF reporting rates. Organizations could also assess
whether implementation has improved patient safety by tracking
Usability testing revealed that users often experienced
the number of usability related design flaws identified that could
increased difficulty when using less frequently used device
have contributed to use errors leading to harm.
 C.J. Flewwelling et al. / Journal of Biomedical Informatics 51 (2014) 80–85 85

6. Limitations in adverse events. Medical equipment maintenance data can be

useful in helping to identify educational opportunities, problems
One significant limitation of the outlined methodology was the with specific devices or other trends that may be of interest.
process used for normalization. The equipment inventory count Although the proposed methodology is limited by the quality of
was used in normalizing the reporting rate as there is presumed an organization’s medical equipment maintenance data, it does
to be a correlation between frequency of device use and equipment provide a relatively efficient, easy to implement and useful process
inventory levels. However, a high inventory level might also be the for identifying devices with a high potential of containing latent
result of other factors, such as a need for the device to be more usability related design faults. Once identified, mitigation strate-
readily available across multiple locations. Other factors for which gies can be employed, such as additional targeted user training
data is not available could have also had an effect, such as whether or abstaining from procuring additional units of the identified
devices are used repeatedly by the same user, and with what fre- devices. It is therefore recommended that the proposed methodol-
quency. Therefore, the accuracy of the results could be improved ogy be applied by healthcare organizations both to reduce the
if the data were normalized using a metric more closely resembling occurrence of use errors, and to reduce the frequency of NFF
frequency of device use. Alternatively, it may be worth investigat- reporting.
ing not normalizing for frequency of use at all. While this would
reduce the likelihood that a usability error was the cause of the References
NFF report, it would allow investigators to focus on those devices
[1] Food and drug administration. Medical device use-safety: incorporating human
with the absolute highest number of NFF reports.
factors engineering into risk management. <http://www.fda.gov/MedicalDevices/
Another limitation was the presence of missing and contradic- DeviceRegulationandGuidance/GuidanceDocuments/ucm094460.htm>; 2000
tory data within the medical equipment maintenance data. In [20.11.12].
[2] Cooper JB et al. Preventable anesthesia mishaps: a study of human factors.
order to generate the list of all NFF reports the failure mode classi-
Anesthesiology 1978;49:399–406.
fication and free text comment fields were queried. In a number of [3] McConnell EA, Cattonar M, Manning J. Australian registered nurse medical
cases one or both fields were left blank at the time of equipment device education: a comparison of simple vs. complex devices. J Adv Nurs
repair. The number of reports with blank fields ranged from 1996;23:322–8.
[4] Food and drug administration. Human factors implications of the new GMP
approximately 0% (NSAHO) to 2% (UHN) across the various hospi- rule. Overall requirements of the new quality system regulations. <http://
tals from which data was collected. In the cases where both fields www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/HumanFactors/
were left blank the report had to be ignored as it was unknown ucm119215.htm>; 1988 [20.11.12].
[5] Norris B. Human factors and safe patient care. J Nurs Manage 2009;17:203–11.
whether or not a fault had been identified. Furthermore, comments [6] Draper S. Human factors engineering a partnering opportunity for clinical
were occasionally included in the free text field that contradicted engineering. J Clin Eng 2004;29(4):198–205.
the fault selected in the failure mode field. Because there was no [7] Draper S, Nielsen G, Noland M. Using no fault found in infusion pump
programming as a springboard for learning about human factors engineering.
way to determine which field contains the accurate information, Jt Comm J Qual Saf 2004;30(9):515–20.
all reports containing contradictions had to be removed. [8] Zhang J, Johnson TR, Patel VL, et al. Using usability heuristics to evaluate
patient safety of medical devices. J Biomed Inform 2003;36:23–30.
[9] Kushniruk A, Patel V. Cognitive and usability engineering methods for the
evaluation of clinical information systems. J Biomed Inform 2004;37:56–76.
7. Conclusions [10] Food and drug administration. Medical device user-safety: Incorporating human
factors engineering into risk management. http://www.fda.gov/downloads/
Medical device designs are becoming more complex, incorpo- MedicalDevices/DeviceRegulationsandGuidance/GuidanceDocuments/ucm094461.
pdf>; 2000 [20.11.12].
rating diverse functions and connecting with increasingly more
[11] Cassano-Piché A, Cafazzo JA, Chagpar A, et al. Choosing safer medical
products and environments. As discussed at the beginning of this technologies: how human factors methods can help in the procurement
article, latent usability related design faults are often implicated process. Biomed Instrum Technol 2010;44:49–56.