Generating Options for Active Risk Control (GO-ARC): Introducing a Novel Technique

Generating Options for Active Risk Control (GO-ARC): Introducing a Novel Technique Authors: Alan J. Card, James R. Ward, P. John Clarkson University of Cambridge Engineering Design Centre This is a pre-print copy. It may differ from the final published version, available at: http://onlinelibrary.wiley.com/doi/10.1111/jhq.12017/abstract Please cite this work as: Card, A. J., Ward, J. R. and Clarkson, P. J. (2013), Generating Options for Active Risk Control (GOARC): Introducing a Novel Technique. Journal for Healthcare Quality. doi: 10.1111/jhq.12017 ABSTRACT Background: After investing significant amounts of time and money in conducting formal risk assessments such as root cause analysis (RCA) or failure mode and effects analysis (FMEA), healthcare workers are left to their own devices in generating high quality risk control options. They often experience difficulty in doing so, and tend toward an overreliance on administrative controls (the weakest category in the hierarchy of risk controls). This has important implications for patient safety and the cost-effectiveness of risk management operations. This paper describes a before-and after pilot study of the Generating Options for Active Risk Control (GO-ARC) Technique, a novel tool to improve the quality of the risk control options generation process. Outcome Measures: The quantity, quality (using the three-tiered hierarchy of risk controls), variety, and novelty of risk controls generated. 1 Results: Use of the GO-ARC Technique was associated with improvement on all measures. Conclusions: While this pilot study has some notable limitations, it appears that the GO-ARC Technique improved the risk control options generation process. Further research is needed to confirm this finding. It is also important to note that improved risk control options are a necessary, but not sufficient, step toward the implementation of more robust risk controls. BACKGROUND In recent years, root cause analysis (RCA),(1–5) failure mode and effects analysis,(6,7) and other formal risk assessment techniques (7–10) have been adopted by the healthcare sector to improve the quality of risk management and patient safety operations. These tools are intended to assist users in the problem-focused task of risk assessment (i.e., identifying, characterizing & estimating, and making determinations about the need to take action to control risks).(11) None of them directly support the solutions-focused task of risk control (also called risk treatment), through which action (in the form of risk controls) is taken to reduce or eliminate negative risks. Risk control is informed by the outputs of the risk assessment process, but a thorough understanding of the problem(s) to be solved does not necessarily translate directly to successful solutions.(2) In the industrial settings where these techniques originated, risk assessment is commonly led by safety / reliability engineers, who have extensive training in converting requirements (such as those identified through risk assessment) into robust and effective interventions.(12) Healthcare workers, however, typically lack such training.(13,14) Perhaps as a result, they often experience 2 difficulty in generating risk control options or assessing the quality of those they have devised.(2,13,15–17) Given the problem-solving objective of the risk control process, improved outcomes (e.g., reductions in surgical site infections or door-to-balloon time for heart attack patients) are, of course, the most important measure of risk control quality, but outcome data is not available during the process of generating and assessing risk control options (i.e., prior to implementation). A process measure of risk control quality is therefore required to guide this stage of practice. The broader risk management community has developed a concept that is well suited for this purpose: the hierarchy of risk controls.(2,18) According to Manuele,(18) who describes a number of versions of the hierarchy: …a hierarchy of controls establishes the actions to be considered in an order of effectiveness to resolve unacceptable hazardous situations. …For many situations, a combination of the risk management methods included in a hierarchy of controls may be applied. However, the expectation is that sequential consideration will be given to each method in a descending order, and that reasonable attempts will be made to eliminate or reduce the hazards and their associated risks by taking the more effective steps higher in the hierarchy before lower steps are considered. A lower step is not to be chosen until practical applications of the preceding higher levels are exhausted. (p.35) We have previously described the three-tiered hierarchy of risk controls described below: Elimination, the most robust tactic, means eliminating the hazard (source of harm) or the target (entity at risk). This can mean transferring the risk to another entity, substituting a less hazardous process, material, etc., or discontinuing the hazardous process entirely. 3 Design Controls (also known as engineering controls) are the next most robust approach. They improve safety without relying on people to take prescribed actions. They may include physical barriers, failsafe design, automation with forcing functions, usability improvements, etc. Administrative Controls rely on people to take prescribed actions (or prompt them to do so). These may include policies/procedures, training, signage, alarms, automation without forcing functions, etc. Despite being the least robust of the three, administrative controls are by far the most commonly used approach in healthcare risk management.(2) RATIONALE Despite the adoption of time-intensive risk assessment techniques, current practice in healthcare risk management has not yet managed to achieve any measurable improvement in patient safety at the systems level.(19) It may be worth bringing healthcare risk management practice in line with best practice in other industries by reducing its heavy reliance on administrative controls. This study describes a novel tool to assist healthcare personnel in generating high quality risk control options as measured by the hierarchy of risk controls: The Generating Options for Active Risk Control (GO-ARC) Technique. Drawing on examples from the analogous process of prospective risk identification, (7,8,20,21), the GO-ARC Technique was designed as a structured brainstorming technique in which a series of prompts is used to elicit risk control options. Each prompt is a risk control strategy. They are presented one-by-one, along with a brief just-in-time training element consisting of a definition and illustrative examples, and participants are asked to brainstorm ways of applying each to the risk at hand. 4 As shown in Table 1, three of the prompts are the tiers of the hierarchy of risk controls described above (i.e., Elimination, Design Controls, and Administrative Controls), and the remaining two are Detection / Situational Awareness and Preparedness. The first is based on the notion, familiar to many users of the failure mode and effects analysis (FMEA) technique,(6,22) that the ability to detect that something is going wrong before it causes harm (or before it causes as much harm as it could) reduces risk substantially. Clinical alarms are a good example. Preparedness focuses on ensuring the ability to mount an effective response if impending or actual harm is detected.(23) An example of this would be ensuring that appropriately matched blood is available for a patient before starting surgery, if significant blood loss is expected. The risk management community views risk as the product of likelihood and severity. Where the first three prompts primarily address likelihood, detection/situational awareness and preparedness primarily address severity, with the aim of reducing risk by increasing resilience.(24) Table 1. Prompts Used in the GO-ARC Structured Brainstorming Technique GO-ARC Prompts Elimination Design Controls Administrative Controls Detection / Situational Awareness Preparedness 5 Here we present a before-and-after pilot study examining the use of the GO-ARC Technique, and assess the resulting risk control options in terms of four variables suggested by the design research literature, namely: quantity, quality, variety and novelty.(25) METHODS Approximately 60 attendees of the educational meeting of a state-level professional organization for healthcare risk managers participated in an uncontrolled before-and-after study aimed at empirically demonstrating the value of taking a structured approach to risk control option generation. They were presented with the scenario of a patient safety incident as described through a risk assessment using the “5 Whys” technique.(26) The scenario is described below: Risk: Inpatient suicide •Why: Patient hung himself from pipes beneath the sink of a public restroom near the dialysis unit. •Why: Patient absconded from psychiatric unit. •Why: Patient walked out the door of the psychiatric unit right behind a patient transportation clerk who was exiting the unit. The patient transportation clerk noticed that the patient was leaving, but assumed he was allowed to because no one from the unit told the patient to stop. •Why: The patient care assistant at the front desk saw the patient leaving, but assumed the patient was going with the patient transportation clerk to attend an appointment elsewhere in the hospital. 6 •Why: Poor communication between the patient transportation clerk and the patient care assistant, both of whom thought the other was responsible for the patient. Participants were first asked to brainstorm risk control options as they normally would. After these risk control options were recorded, participants were then led through the GO-ARC structured brainstorming Technique. They were introduced to the five prompts one by one, and asked to brainstorm ways of applying each approach in turn. The quantity of risk controls generated was measured by counting. The results were recorded in real time and later coded according to the three-tiered hierarchy of risk controls (2) as a measure of quality, and according to design principles arising out of the results themselves, as a measure of variety and novelty. Variety refers not to individual ideas, but rather to groups of ideas that correspond to different functional principles for solving a given problem (e.g., physical barriers or automation with forcing functions).(27) Functional principles were first identified at a high level (by coding the options according to the hierarchy of risk controls), and then at a more detailed level as strategies for achieving elimination, design control, or administrative control emerged from the results. Novelty refers to the degree to which a given design (risk control option) is outside what would otherwise be expected.(27) RESULTS Use of the GO-ARC Technique took approximately 10-15 minutes and generated a greater number of risk control options overall, a greater number non-administrative risk controls, and a higher proportion of non-administrative risk controls, as compared to current practice. It also led to notable increases in the variety and novelty of risk control options generated. 7 Quantity and Quality As shown in Table 2, unstructured brainstorming resulted exclusively in administrative controls, and calls for additional risk assessment (which in and of itself does nothing to control risk). Table 2. Risk Control Options Generated Using Current Practice Risk Control Elimination Design Control Option Administrative Not a Risk Control Control More O communication Handoff O communication Clearer job O safety analysis Patient flow O review Improved O patient assessment Alarm system O Promoting a O culture of speaking up Total = 7 0 0 5 2 Proportion 0 0 0.71 0.29 Table 3 shows the risk control options generated in response to each of the five prompts in the GO-ARC Technique. The GO-ARC Technique resulted in 3.6 times the number of unique genuine risk controls to choose from (18 vs. 5), and the proportion of unique non-administrative controls generated using the GO-ARC Technique was 25%, compared to 0% for current practice. 8 Table 3. Risk Control Options Generated Using the GO-ARC Technique Prompt Risk Control Option Elimination Design Admini- Not a Risk Control strative Control (X) Control or is a Repeat (/) Elimination Patients don’t leave the X unit (bring all services to them) Close psychiatric unit X Don’t admit suicidal X patients Design Banding the patient X (tied to an automatic alarm system) Automated access X control system Cover pipes X Environmental X assessment of risk Admin Education around X communication Patient transport X checkout procedure Policies X Detection / Alarms tied to patient Situational Surveillance cameras X Awareness Frequency of X / observation 9 Staff awareness of X individual patient needs (and who the patients are) Patient uniforms X Increased awareness X during visiting hours GPS or RFID tracking X Pre-shift briefs, X checklists to establish situational awareness Prepared- Drills for missing ness psychiatric patients X Develop a code to call X when psychiatric patients go missing (e.g., Code Orange) Total = 20 3 2 13 2 Proportion 0.15 0.10 0.65 0.1 Variety and Novelty Risk controls generated using the GO-ARC Technique demonstrated greater variety, as shown in Table 4. The GO-ARC results covered 12 different functional principles, while unstructured brainstorming resulted in options covering only five. Use of the GO-ARC Technique also led to increased novelty; it resulted in risk control options that employed 7 functional principles that did not arise of using current practice. Use of current practice did not result in the generation of risk control options using any functional principles that were not also seen in the GO-ARC results. 10 More communication O Handoff communication O Improved patient O O assessment Alarm system O O Promoting a culture of O speaking up Patients don’t leave the unit X (bring all services to them) Close psychiatric unit X Don’t admit suicidal X patients Banding the patient (tied to X X an automatic alarm system) Automated access control X X system Cover pipes X Education around X communication Patient transport checkout X 11 Other Checkout Risk Control Options Patient observation Procedures) Finding absconded patients Controls Staff-staff communication Controls Patient risk assessment Admin Controls (Policies and Prompts Admin Automation w/ forcing functions Principles Design Physical barriers Functional Eliminating the target Primary Eliminating the hazard Elimination Automation w/o forcing functions Table 4. Variety and Novelty of Risk Control Options Generated procedure Policies X Surveillance cameras X X Frequency of observation X Staff awareness of X X individual patient needs (and who the patients are) Patient uniforms X X Increased awareness during X visiting hours GPS or RFID tracking X Pre-shift briefs, checklists X X X X to establish situational awareness Drills for missing X X psychiatric patients Develop a code to call X X when psych patients go missing (e.g., Code Orange) O = Generated using current practice; X = Generated using the GO-ARC Technique DISCUSSION Brainstorming Devising risk controls is essentially a design process, in which the product is one or more interventions intended to reduce risk. The design research community has developed measures for the performance of design ideation (i.e., options generation) techniques that might usefully be applied to the analysis of the brainstorming portion of this study. Suggested measures include quantity, quality, variety and novelty.(25) 12 Quantity The GO-ARC Technique resulted in the generation of 3.6 times the number of unique genuine risk controls to choose from (18 vs. 5). By itself, giving users more risk control options to choose from does not necessarily do much to improve risk control. If the additional risk control options are little different from those that would otherwise be generated, it may simply represent a waste of time. And if the additional risk control options are mostly of a lower quality than would otherwise be generated, it might actually increase the likelihood of poor risk controls being chosen (i.e., it would make things worse). If, however, the additional risk control options are sufficiently different from what would otherwise be generated, and also of higher quality, this increase in quantity should benefit users. Quality The GO-ARC Technique was specifically designed to improve the quality of risk control options generated as measured by the three-tiered hierarchy of risk controls.(2) According to this hierarchy, the most robust response is eliminating the hazard or target, followed by design controls (i.e., interventions such as physical barriers or forcing functions that do not rely on people taking prescribed action), while administrative controls (i.e., interventions such as training or policies that rely on people taking prescribed action) are considered the least robust.(2) Non-risk controls (e.g., calls for additional risk assessment, repeated calls for the same risk control) are not considered to have any direct effect. The number of unique non-administrative controls generated using the GO-ARC Technique was 5, equal to the total number of genuine risk controls generated using current practice. In terms of percentages, it led to 25% non-administrative controls, in comparison to 0% from current practice (5/20 vs. 0/7). The proportion of non-risk controls generated was also lower by about 13 two-thirds. Figure 2 compares the distribution of risk control options generated using the two approaches. Figure 1. Risk Control Options by Technique and Classification on the Hierarchy of Risk Controls The design literature also calls for a measure of feasibility in assessing quality, but the GO-ARC Technique is not designed to address this. It has grown out of the process of developing a broader Active Risk Control (ARC) Toolkit, which includes a separate tool for analyzing and evaluating the risk controls generated using the GO-ARC component. Variety Variety refers not to individual ideas, but rather to groups of ideas that correspond to different general approaches to solving a given problem (i.e., categories of design approaches).(27) As Table 5 demonstrates, use of the GO-ARC Toolkit resulted in approximately 2 ½ times the 14 number of design principles employed, meaning that it resulted in 2 ½ times the level of variety. Even (perhaps especially) when quality is increased, additional variety is important. If all the additional high-quality risk control options use the same functional principle, they might be too similar for the marginal benefit of implementing more than one to be worthwhile. Thus, they could be seen as adding only one genuinely new approach. In light of the findings on quantity and quality, the sharp increase this study demonstrated in the number of functional principles elicited using the GO-ARC Technique (i.e., the increased variety) means that decision-makers will have a greater number of distinct, high-quality options to choose from. Novelty Novelty refers to the degree to which a given design (risk control option) is outside what would otherwise be expected.(27) Use of the GO-ARC Technique resulted in the generation of 13 risk control options using 7 functional principles that were not employed in the options generated using current practice, as well as 5 risk control options covering all 4 of the functional principles that resulted from current practice. Thus there was a marked increase in novelty based on the introduction of new functional principles. Interestingly there was little increase in novelty within functional principle categories (i.e., for those categories which overlapped, the risk control options generated were very similar). This may be an artifact of the study design. Additional research will be required to determine whether this functional principle-dependent novelty is a general feature of the Technique. Limitations A number of potentially significant limitations apply to this study. This was a single uncontrolled before-and-after study, which may not be generalizable. In particular, it was conducted with an unusually large and experienced group of contributors, and it is unclear to 15 what degree the results would apply in a more traditional setting. However, it is quite likely that a formal design process tool that improves the work of experts would be even more effective among less experienced users (see for instance Winkleman & Hacker (28)). The fact that the technique’s inventor was the facilitator for the brainstorming study could also be a source of bias, either because participants felt an internal impetus to try harder during the treatment (GO-ARC) phase, or because the facilitator unintentionally encouraged them to do so. It is also possible that some experience bias was involved, that the first, unstructured, brainstorming session served as practice for the GO-ARC session, leading to improved performance. However, because the participant population was made up primarily of experienced healthcare risk managers, this seems unlikely to have had a major impact. The categories used to assess variety and novelty were derived from the responses on a post hoc basis. It is possible that pre-developed categories might have resulted in different findings in terms of degree, though not in kind. There were 3 pre-developed categories for quality, in the form of the hierarchy of risk controls, and even if the analysis were restricted to these 3 extremely high-level categories, the findings would still support improved variety and novelty. Finally, the GO-ARC Technique cannot solve the problem of weak risk controls by itself. As a tool for generating risk control options, its goal is simply to ensure that users generate more robust risk control options for consideration, and in this it seems successful. But while having a more robust pool of options to choose from is a necessary step, it is not a sufficient one, toward the actual implementation of more effective risk controls. The technique seems likely to have at least some positive effect, but it is entirely possible for users to ignore higher quality options and continue to implement exclusively administrative controls. (In a recent systematic review of risk 16 control after root cause analysis, we found that half of the studies describing the risk controls implemented used administrative controls alone.(2)) Further, not every ‘strong’ risk control will actually be higher-quality in terms of cost-effectiveness. It is important to note, however, that risk controls that do not work are inherently not cost-effective (they represent pure waste) and that weak risk controls may often appear more cost-effective in the short term than they prove in the long term, due to low implementation costs, but high costs for sustainment (if they are actually sustained).(29) Ensuring that the GO-ARC Technique lives up to its potential may therefore require that it be coupled with additional tools and organizational processes to analyze and evaluate risk control options (analogous to the risk analysis and risk evaluation components that follow risk identification as parts of the risk assessment process(11)). This function lies beyond the scope of an options generation technique and will require an assessment of a wider set of variables, including the expected benefits, resource requirements, and secondary risks,(11) and whether risk control options are implementable and tied by a logic model to the intended improvement,(30,31) among others. We have worked to develop such tools as part of a broader Active Risk Control (ARC) Toolkit. In future work, we plan to describe an evaluation of the complete Toolkit, which will place the GO-ARC Technique in the context of this overarching process. It is important to note, however, that with or without such analytical tools, the quality of risk controls implemented is inherently limited by the quality of the risk control options generated for consideration; thus, the GO-ARC Technique has the potential to add value in either case. 17 CONCLUSIONS We described the Generating Options for Active Risk Control (GO-ARC) Technique, a novel tool for generating risk control options in response to an assessed risk. We also reported a pilot study that provides the first evaluation of the technique’s effectiveness. We found that the GOARC Technique required very little time to use, and led to marked improvements on all measures of design ideation performance. Further research is needed to confirm these results, but based on this study it appears that the GO-ARC Technique improves risk control option generation. REFERENCES 1. Bagian JP, Lee C, Gosbee J, DeRosier J, Stalhandske E, Eldridge N, et al. Developing and deploying a patient safety program in a large health care delivery system: you can’t fix what you don't know about. Jt Comm J Qual Improv. 2001;27(10):522–32. 2. Card AJ, Ward J, Clarkson PJ. Successful risk assessment may not always lead to successful risk control: A systematic literature review of risk control after root cause analysis. Journal of Healthcare Risk Management. 2012;31(3):6–12. 3. The Joint Commission. Sentinel Events. The Joint Commission. Oakbrook Terrace, IL; 2007. p. 13. 4. National Patient Safety Agency. Root cause analysis (RCA) tookit [Internet]. 2004 [cited 2012 Aug 17]. Available from: http://www.nrls.npsa.nhs.uk/resources/collections/rootcause-analysis/?entryid45=59901 5. Taitz J, Genn K, Brooks V, Ross D, Ryan K, Shumack B, et al. System-wide learning from root cause analysis: a report from the New South Wales Root Cause Analysis Review Committee. Quality & Safety in Healthcare. 2010 Jul;19(6):e63. 18 6. DeRosier J, Stalhandske E, Bagian JP, Nudell T. Using health care failure mode and effect analysis: The VA National Center for Patient Safety’s prospective risk analysis system. Joint Commission Journal on Quality Improvement. 2002;28(5):248–67. 7. Ward J, Clarkson J, Buckle P, Berman J, Lim R, Jun T. Prospective Hazard Analysis: Tailoring Prospective Methods To A Healthcare Context [Internet]. 2010. Available from: http://www.haps2.bham.ac.uk/publichealth/psrp/PS035_Project_Summary.shtml 8. Card AJ, Ward JR, Clarkson PJ. Beyond FMEA: The structured what-If technique (SWIFT). Journal of Healthcare Risk Management. 2012;31(4):23–9. 9. Card AJ, Harrison H, Ward J, Clarkson PJ. Using prospective hazard analysis to assess an active shooter emergency operations plan. Journal of Healthcare Risk Management. 2012;31(3):34–40. 10. Hyman WA, Johnson E. Fault tree analysis of clinical alarms. Journal of Clinical Engineering. (Hyman, Johnson) Department of Biomedical Engineering, Texas A and M University, College Station, TX, United States; 2008;33(2):85–94. 11. ISO. ISO 31000: Risk management — Principles and guidelines on implementation. Event (London). Geneva; 2009. 12. Engineering Accreditation Commission. Criteria for accrediting engineering programs. Baltimore, MD; 2011. 13. Pham JC, Kim GR, Natterman JP, Cover RM, Goeschel CA, Wu AW, et al. ReCASTing the RCA: an improved model for performing root cause analyses. American Journal of Medical Quality : 2010 Jan;25(3):186–91. 14. Youngson GG, Flin R. Patient safety in surgery: non-technical aspects of safe surgical performance. Patient Safety in Surgery. BioMed Central; 2010 Jan;4(1):4. 19 15. Percarpio KB, Watts BV, Weeks WB. The effectiveness of root cause analysis: what does the literature tell us? Joint Commission Journal on Quality & Safety. 2008 Jul;34(7):391– 8. 16. Wallace LM, Spurgeon P, Earll L. Evaluation of the NPSA 3 day root cause analysis training programme: Final report (Revised October 2006). 2006. 17. Wu AW, Lipshutz AKM, Pronovost PJ. Effectiveness and efficiency of root cause analysis in medicine. JAMA. 2008;299(6):685–7. 18. Manuele FA. Risk assessment and hierarchies of control. Professional Safety. 2005;50(5):33–9. 19. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. NEJM. 2010 Nov;363(22):2124–34. 20. Maguire R. Safety cases and safety reports: Meaning, motivation and management. Burlington, VT: Ashgate Publishing; 2006. p. 101–3. 21. Pronovost PJ, Lilford R. A Road Map For Improving The Performance Of Performance Measures. Health Affairs. 2011;30(4):569–73. 22. ISO. ISO 31010: Risk management — Risk assessment techniques. Event (London). Geneva; 2009. 23. Keim ME. Building human resilience: the role of public health preparedness and response as an adaptation to climate change. American Journal of Preventive Medicine. 2008 Nov;35(5):508–16. 20 24. Habraken M, Van der Schaaf T. If only....: failed, missed and absent error recovery opportunities in medication errors. Quality & Safety in Health Care. 2010 Feb;19(1):37– 41. 25. Shah J, Kulkarni S. Evaluation of idea generation methods for conceptual design: Effectiveness metrics and design. Journal of Mechanical Design. 2000;122(December):377–84. 26. Serrat O. The five whys technique. International Publications. Paper 198. Washington, DC; 2010 p. 1–3. 27. Shah JJ, Smith SM, Vargas-Hernandez N. Metrics for measuring ideation effectiveness. Design Studies. 2003 Mar;24(2):111–34. 28. Winkelmann C, Hacker W. Question-answering-technique to support freshman and senior engineers in processes of engineering design. Journal of Technology and Design Education. 2010;20:305–15. 29. NIOSH. NIOSH topic: Engineering controls [Internet]. 2010 [cited 2012 Sep 8]. Available from: http://www.cdc.gov/niosh/topics/engcontrols/ 30. Iedema R, Jorm C, Braithwaite J. Managing the scope and impact of root cause analysis recommendations. J Health Organ Manag. 2008;22(6):569–85. 31. Shekelle P, Pronovost P, et al. Advancing the Science of Patient Safety. Annals of Internal Medicine. 2011;154(10):693–6. 21