Research and applications Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems David C Radley,1 Melanie R Wasserman,2 Lauren EW Olsho,2 Sarah J Shoemaker,2 Mark D Spranca,2 Bethany Bradshaw3 1 Institute for Healthcare Improvement, Cambridge, Massachusetts, USA 2 US Health Division, Abt Associates Inc, Cambridge, Massachusetts, USA 3 School of Public Health, Johns Hopkins University, Baltimore, Maryland, USA Correspondence to Dr Lauren Olsho, US Health Division, Abt Associates, Inc, 55 Wheeler St, Cambridge, MA 02138, USA; lauren_olsho@abtassoc.com Received 2 August 2012 Revised 19 December 2012 Accepted 10 January 2013 Published Online First 21 February 2013 ABSTRACT Objective Medication errors in hospitals are common, expensive, and sometimes harmful to patients. This study’s objective was to derive a nationally representative estimate of medication error reduction in hospitals attributable to electronic prescribing through computerized provider order entry (CPOE) systems. Materials and methods We conducted a systematic literature review and applied random-effects metaanalytic techniques to derive a summary estimate of the effect of CPOE on medication errors. This pooled estimate was combined with data from the 2006 American Society of Health-System Pharmacists Annual Survey, the 2007 American Hospital Association Annual Survey, and the latter’s 2008 Electronic Health Record Adoption Database supplement to estimate the percentage and absolute reduction in medication errors attributable to CPOE. Results Processing a prescription drug order through a CPOE system decreases the likelihood of error on that order by 48% (95% CI 41% to 55%). Given this effect size, and the degree of CPOE adoption and use in hospitals in 2008, we estimate a 12.5% reduction in medication errors, or ∼17.4 million medication errors averted in the USA in 1 year. Discussion Our findings suggest that CPOE can substantially reduce the frequency of medication errors in inpatient acute-care settings; however, it is unclear whether this translates into reduced harm for patients. Conclusions Despite CPOE systems’ effectiveness at preventing medication errors, adoption and use in US hospitals remain modest. Current policies to increase CPOE adoption and use will likely prevent millions of additional medication errors each year. Further research is needed to better characterize links to patient harm. which may further reduce errors. There is also some evidence that CPOE may cause errors.5 CPOE’s impact on medication errors and outcomes remains uncertain because of the varied clinical settings, CPOE system origins (commercial vs created in-house), and quality of existing studies.6 With its Healthcare Information Technology for Economic and Clinical Health (HITECH) provision, the American Recovery and Reinvestment Act of 2009 authorized US$20 billion in funding to assist in the development of a robust health information technology (health IT) infrastructure to improve healthcare safety and quality. Among the HITECH Act’s provisions are incentive payments to outpatient physicians and hospitals to support health IT implementation, including CPOE implementation as a core requirement.7 The ultimate goal of CPOE is improved safety, quality, and value of patient care. Medication errors are an important intermediate, measurable outcome in pursuit of that goal. In this investigation, we examine the impact of CPOE on medication error frequency. OBJECTIVE The objective of this study was to provide a baseline national estimate of medication errors averted in hospitals due to use of CPOE, using data on CPOE use in 2008, before implementation of the HITECH Act. The baseline estimate and methodology may be useful to track national progress on CPOE adoption, use, and outcomes, and to inform the evolving federal strategy to build an effective health IT infrastructure. MATERIALS AND METHODS BACKGROUND AND SIGNIFICANCE Open Access Scan to access more free content To cite: Radley DC, Wasserman MR, Olsho LEW, et al. J Am Med Inform Assoc 2013;20:470–476. 470 The Institute of Medicine estimates that, on average, hospitalized patients are subject to at least one medication error per day.1 Medication errors are expensive and sometimes harmful to patients.2 3 The Institute of Medicine estimates that at least a quarter of all medication-related injuries are preventable, and recommends electronic prescribing (e-prescribing) through a computerized provider order entry (CPOE) system as one way to reduce medication errors and patient harm.4 Electronic entry of medication orders through CPOE may reduce errors from poor handwriting or incorrect transcription. CPOE systems often include functionalities such as drug dosage support, alerts about harmful interactions, and clinical decision support, This study was conducted in two phases. First, we developed supporting statistics describing CPOE adoption and implementation, the volume of medication orders processed in our target population of hospitals, the number and proportion of medication orders processed through CPOE, the expected error rate without CPOE, and the expected percentage reduction in medication error rates resulting from CPOE. Second, we used supporting statistics to derive two nationally representative outcome estimates: the percentage and absolute reduction in medication errors in acute-care hospitals over a 1-year period due to CPOE use. Below we describe the data sources and target population for this study, followed by the data elements and analytic techniques used for each statistic. Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241 Research and applications Data sources Study data were drawn from the following: the 2007 American Hospital Association (AHA) Annual Survey (AHA survey); the AHA Hospital Electronic Health Record (EHR) Adoption Survey (EHR survey), collected in 2008 as a supplement to the 2007 AHA survey; the 2006 American Society of Health-System Pharmacists (ASHP) national survey of hospital pharmacies8; and a systematic review of the peer-reviewed literature. Target population Hospitals represented in the AHA survey were included if they provided general or pediatric acute medical and surgical care and self-identified as private-for-profit, private not-for-profit, or public. We excluded long-term care and federally owned hospitals, and hospitals outside the 50 states or the District of Columbia. Our final sample included 4701 hospitals. Supporting statistics CPOE adoption and implementation EHR survey data were used to estimate CPOE adoption and implementation for a nationally representative sample of hospitals. The survey’s methodology has been described previously.9 The response rate from the 4701 eligible hospitals was 60.9% (N=2864), and the response rate to CPOE questions was slightly lower at 60.3% (N=2833). CPOE adoption was estimated using EHR survey items on the presence of CPOE, number of units in which it had been implemented, and its functionalities. A hospital was counted as a CPOE adopter if it had in at least one unit an operational CPOE system capable of processing prescription drug orders. We used regression imputation to estimate CPOE adoption among non-responding hospitals. A logistic regression model identified hospital characteristics associated with CPOE adoption from the AHA survey; estimated model parameters were used to derive the predicted probability of CPOE adoption among non-responding facilities. CPOE implementation was estimated using EHR survey items asking respondents to estimate the fraction of drug orders processed via CPOE. Respondents had five response options: 0%, 1–25%, 26–50%, 51–90%, or 91–100%. Responding hospitals were assigned the mid-point of their reported range as their CPOE implementation measure. Responders’ mean value (58.8%) was imputed to facilities with CPOE not responding to this question. For non-responding facilities, the mean value of 58.8% was multiplied by the predicted probability of having CPOE. Sensitivity analyses to evaluate the impact of this approach are reflected in reported estimate bounds. Medication order volume We used data from the ASHP survey to estimate the volume of medication orders processed each year by US hospitals. The ASHP survey is a nationally representative stratified random sample of hospital pharmacy directors at general and children’s hospitals. Pedersen and colleagues provide details on survey methodology, implementation and results.8 Published results include the average number of prescription drug orders per patient-day for hospitals stratified by bed size. For each targeted hospital, we multiplied the appropriate average prescription-drug-order-per-patient-day statistic by AHA survey estimates of total bed days to estimate the total number of medication orders processed during a year. Number and proportion of medication orders processed through CPOE We then combined hospital-specific estimates of CPOE adoption and implementation from the 2008 EHR survey with estimates of medication order volume from the 2006 ASHP survey to obtain a nationally representative estimate of the total number and proportion of medication orders processed using CPOE. We interpret these composite estimates as reflective of the year 2008, implicitly assuming that 2006 medication order volume estimates did not substantially change over the intervening 2-year period. Expected medication error rate without CPOE and expected reduction in medication error rates resulting from CPOE To date, no nationally representative dataset exists linking CPOE use to medication errors. Thus, we extracted data from a systematic literature review and used meta-analytic random effects techniques to estimate three parameters: medication error rates when CPOE is not used, medication error rates when CPOE is used, and the percentage difference between them. Our literature search strategy was adapted from a review by Ammenwerth and colleagues.6 Study inclusion criteria were as follows: the intervention included e-prescribing functionality; an e-prescribing system was compared with handwritten ordering; the study setting was an inpatient section of an acute-care hospital; providers with prescribing authority were the primary users of the e-prescribing system; the study used an experimental or quasi-experimental design, including randomized controlled trial, non-randomized controlled trial, before–after trials, or repeated time-series analysis; and the mode of medication error detection was consistent before and after intervention. Finally, since our goal was to construct aggregated medication error rates before and after CPOE implementation, we retained only studies that either reported medication error rates per order or provided sufficient data from which to calculate these rates. Studies conducted outside the USA were excluded. Using the search terms of Ammenwerth et al, we updated the search using PubMed in February 2009, identifying 390 studies. Each was reviewed by two study authors (MRW and DCR). After applying the a priori inclusion/exclusion criteria, 10 studies were retained.10–19 Based on later expert reviewer feedback, we eliminated one additional study that solely used a voluntary reporting method for error detection,19 leaving nine studies for our final pooled analysis. The outcome of interest was medication error frequency. For each included study, we used the author’s definition of a medication error, encompassing errors in ordering, transcribing, dispensing, administration, and monitoring.10 11 Some reviewed studies also reported adverse drug events (ADEs) and potential adverse drug events (PADEs). These studies were not excluded from our literature review, but we used information provided in reviewed studies to exclude ADEs and PADEs from data aggregation procedures whenever feasible, since both are considered distinct from more broadly defined medication errors. Relevant data elements were extracted for pre- and postintervention periods from each study. These included duration (in months) of pre- and post-intervention periods, and numbers of prescription drug orders and medication errors recorded. For each study, we calculated pre- and post-CPOE medication error rates per order per month. These rates were used to calculate the percentage change in medication error rates. Pooled summary statistics were calculated as the weighted average of pre-CPOE medication error rates, and the weighted average Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241 471 Research and applications difference between pre- and post-CPOE medication error rates. Cochran’s Q was significant ( p<0.001), indicating heterogeneity across the nine studies used to calculate the pooled effect. To accommodate within- and between-study variance, we therefore used the DerSimonian–Laird random effects model to pool data.20 Outcome statistics The percentage reduction in medication error frequency due to CPOE is the product of the expected percentage reduction in medication error rates resulting from CPOE, as estimated from our systematic literature review using meta-analytic techniques, and the proportion of medication errors processed through CPOE in hospitals that have adopted and use CPOE, as calculated using estimates from nationally representative surveys described above. We assumed a 0% reduction in medication error rates due to CPOE for orders processed in settings without CPOE. The absolute reduction in medication error frequency due to CPOE is the product of the first outcome statistic, the expected percentage reduction in medication error rates resulting from CPOE, and the expected total number of medication errors if no CPOE were used (ie, total number of medication orders processed during a year multiplied by the expected medication error rate without CPOE), as calculated using estimates from nationally representative surveys as described above. Input supporting statistics were annualized to facilitate calculation of outcome statistics. These outcome statistics thus represent estimated medication error reduction during a 1-year period based on observed levels of CPOE adoption and use in 2008, medication order volume in 2006, and error rates and estimated reductions from our meta-analysis. Varying assumed levels of CPOE adoption and use from observed 2008 levels allows us to extrapolate to expected reductions over a 1-year period as adoption and use increase. These calculations assume that the effect of CPOE adoption and use on medication error rates remains constant as the number of hospitals adopting and implementing CPOE increases. This assumption may not hold if later CPOE adopters differ systematically from earlier adopters, or if medication orders currently processed through CPOE differ systematically from medication orders not currently processed through CPOE. We also assumed that CPOE adoption does not change the total volume of medication orders processed in hospitals, a simplifying assumption necessary for converting the estimated percentage reduction in medication errors into an estimated absolute reduction. More generally, our estimates proceed by applying estimated parameters from our meta-analysis to nationally representative survey measures. This approach implicitly assumes that the estimated error rates from the literature generalize to US hospital settings as a whole. Estimate bounds Constructing probability-based CIs around these outcome statistics was not feasible because of non-independence between several sources of variability embedded in supporting statistics. Sources of variability included within- and between-study variation in our meta-analysis, sampling variance on prescription drug order estimates from the ASHP survey, and imputation procedures used to estimate CPOE adoption and implementation. Instead of probability-based CIs, we therefore constructed logical estimate bounds based on reasonable assumptions about underlying variability in these measures. 472 Our approach for pooling data from the systematic literature review allowed us to derive a probability-based CI around our estimate of the average percentage reduction in medication error rates due to CPOE. In addition, sampling variance on medication order estimates from the ASHP survey influenced our estimates for the total number of medication orders and the number of medication orders processed with CPOE. ASHP-reported SEs on mean medication orders per bed-day were used to calculate probability-based lower and upper bounds on estimates of medication order volume. There was no credible approach for deriving probability-based estimates of variance for measures of CPOE adoption and implementation. Instead, we set logical bounds around each by imposing assumptions about CPOE use for hospitals not responding to the EHR survey. The lower bound assumed that no non-responding hospitals adopted CPOE, and the upper bound assumed that all non-responding hospitals adopted CPOE. Each supporting statistic described above was recalculated using lower and upper bounds as inputs. Upper- and lower-bound supporting statistics were in turn used as inputs to calculate upper- and lower-bound summary statistics. Our intentionally conservative approach resulted in point estimates with relatively wide bounds. Note, however, that, as these are not probability-based CIs, the point estimates do not have an equal probability of taking on any value between reported bounds. Instead, the point estimates give a reasonable approximation of the true value, while the bounds represent possible extreme values that could have been derived from given inputs. RESULTS CPOE use Approximately 34% (1589 of 4701) of US acute-care hospitals had adopted CPOE in 2008. Among the 2833 hospitals responding to the EHR survey, larger hospitals (≥400 beds) were more likely to have adopted CPOE (56%) compared with medium-sized or small hospitals (35% and 30%, respectively). CPOE adoption was more common among urban hospitals (41% vs 28% among rural hospitals, p<0.001) and major teaching hospitals (53% vs 32% in non-teaching hospitals, p<0.001). CPOE adoption was higher among private not-for-profit hospitals (37%) compared with public hospitals (31%) and private for-profit hospitals (32%). CPOE adoption did not significantly differ between independent and health system-affiliated hospitals (34% vs 36%, p=0.13). Table 1 summarizes CPOE adoption in 2008 by hospital characteristic. Table 2 describes 2008 CPOE implementation, the proportion of prescription drug orders processed using CPOE in hospitals with a CPOE system. Many CPOE adopters (39.0%) indicated a very high degree of implementation (>90% of orders processed by CPOE). Still, 42.4% of responding hospitals using CPOE reported <50% implementation. Assuming the midpoint value for each range, and averaging across all hospitals providing CPOE data, mean CPOE implementation was 58.8%. Among hospitals adopting CPOE, there was no statistically significant association between bed size and CPOE implementation levels. Total annual medication order volume was estimated at 1 757 886 464 orders per year based on the 2007 AHA survey and the 2006 ASHP survey. Factoring this with CPOE adoption and implementation in 2008, and extrapolating to all US acutecare hospitals, we estimate that ∼26.1% (bounds 16.0–53.6%) of medication orders in acute-care hospitals were processed using CPOE. Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241 Research and applications medication error rate per order of 0.079 without CPOE. Medication error rates were ∼48% (95% CI 41% to 55%) lower after CPOE implementation (table 4). Table 1 Computerized provider order entry (CPOE) adoption by hospital characteristic, 2008 Characteristic All acute hospitals Pediatric specialty hospitals Bed size Small (6–99) Medium (100–399) Large (≥ 400) Census region* Northeast Midwest South West Ownership type Public Not-for-profit For-profit Member of a health system No Yes Location Rural Urban Teaching status Non-teaching hospital (0 full-time residents) Minor teaching hospital (between 1 and 20 full-time residents) Major teaching hospital (more than 20 full-time residents) With CPOE Without CPOE p Value 992 (35) 14 (100) 1841 (65) (0) NA NA 389 (30) 430 (35) 173 (56) 910 (70) 795 (65) 136 (44) 173 298 356 141 224 633 639 297 (44) (32) (36) (32) (56) (68) (64) (68) <0.001 0.001 211 (31) 670 (37) 111 (32) 471 (69) 1131 (63) 239 (68) 452 (34) 540 (36) 894 (66) 947 (64) 0.128 360 (28) 632 (41) 928 (72) 913 (59) <0.001 705 (32) 1505 (68) 98 (37) 169 (63) 189 (53) 167 (47) 0.005 Summary and outcome statistics are summarized in table 4. At the rate of CPOE adoption and implementation in 2008, our findings suggest that medication errors were reduced by ∼12.5% (bounds 10.6–14.4%). This equates to ∼17.4 million (bounds 0.09–27.1 million) fewer medication errors over a 1-year period than would be expected without CPOE. Figure 1 illustrates observed (under the analytic assumptions of this study) and potential reductions in medication errors during a 1-year time frame associated with varying levels of CPOE adoption. The y-axis is the number of averted errors, and the x-axis is the proportion of hospitals adopting CPOE. The slope of the line for each wedge represents CPOE implementation ( proportion of drug orders processed using CPOE). The black wedge represents medication error reductions associated with CPOE use in 2008. If all US hospitals adopted CPOE, assuming constant implementation levels of ∼60%, as many as 51 million medication errors per year could be averted compared with what would have been expected without CPOE (hatched area). If the proportion of medication orders processed using CPOE were to increase in hospitals that already have CPOE in place (grey shaded area), as many as 36 million errors per year might be averted. Finally, if all hospitals were to use CPOE to process all medication orders, as many as 104 million medication errors per year could be averted (dotted area). <0.001 DISCUSSION Values are number (%). Data in this table are aggregated only from the 2833 hospitals that provided responses to the EHR adoption database supplement questions regarding CPOE adoption. *Geographic region was missing for 72 hospitals. EHR, electronic health record. Expected medication error rate without CPOE and expected change in error rate associated with CPOE use Table 3 summarizes literature review findings. Among the nine pooled studies, eight found a decrease in medication error frequency after CPOE implementation. The last, Walsh et al,18 reported an increase (23%) in medication errors. Pooling data across these studies, we find an expected Table 2 CPOE implementation among hospitals that report having a CPOE system according to hospital size, 2008 Proportion of medication orders processed using CPOE Hospital size, beds 1–25% 26–50% 51–90% ≥ 90% <200 200–299 300–399 ≥ 400 Total (row %) 110 28 22 45 205 46 (13) 12 (12) 10 (14) 16 (11) 84 (12) 78 20 13 16 127 126 42 24 74 266 (31) (27) (32) (30) (30) Percentage and absolute reduction in medication errors (22) (20) (19) (11) (19) (35) (41) (35) (49) (39) Values are number (%). These data represent reported CPOE implementation among the 682 hospitals that indicated having a CPOE system in place and gave responses to CPOE implementation questions in the EHR survey. CPOE, computerized provider order entry; EHR, electronic health record. To our knowledge, our study represents the first attempt to generate a nationally representative estimate of the effect of CPOE on medication error frequency. Our findings suggest that CPOE can substantially reduce medication errors in hospitals. In 2008, ∼34% of US acute-care hospitals had adopted CPOE capable of processing prescription orders. At these adoption and implementation levels, we estimate 17.4 million medication errors per year avoided due to CPOE—a 12.5% reduction nationally. Given the modest adoption and implementation rates to date, there is still great potential for this technology to reduce medication errors. The projected reduction in medication errors represents an important intermediate indicator of potential gains as health IT systems are expanded and more deeply integrated in care delivery systems nationwide. However, it is unclear whether reduced medication errors would translate into reduced patient harm from medications. Several studies reviewed for this analysis provide insight into the potential for harm reduction. Bates et al,10 Evans et al,14 and Igboechi et al15 all report fewer ADEs and fewer serious medication errors with potential to harm patients after CPOE implementation. However, insufficient information was available to construct meta-analytic estimates of CPOE impacts on serious medication errors, PADEs, or ADEs. While our conclusions reflect the fact that the majority of peerreviewed studies find reductions in medication errors associated with CPOE, a few studies have found increases. As noted above, Walsh et al18 found that error rates increased from 18 to 22 per 1000 medication orders after CPOE adoption, although this difference was not statistically significant, and the authors also found fewer non-intercepted serious medication errors and fewer PADEs in the post-CPOE implementation period. Bradley and colleagues illustrate the risk of unintended consequences from CPOE: while Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241 473 Research and applications Table 3 Summary data from systematically peer-reviewed literature evaluating medication error frequency before (pre) and after (post) implementation of computerized provider order entry (CPOE) Duration (months) Pre Author (year of publication) CPOE implementation and study setting (hospital department) Bates et al (1999)10 Select medical and intensive care units (inpatient) Emergency department NICU Intensive care unit Hospital wide (inpatient) 2 6 24 24 Pediatric oncology unit Hospital wide (inpatient) NICU NICU, PICU, select pediatric medical and surgical units (inpatient) Bizovi et al (2002)12 Cordero et al (2004)13 Evans et al (1998)14 Igboechi et al (2003)15 Kim et al (2006)16 Mahoney et al (2007)17 Taylor et al (2008)11 Walsh et al (2008)18 1.7 Post 5.6 Medication orders Medication errors Rate per 1000 orders Pre Pre Post Pre Post Post Percentage difference (unweighted)* 10070 42516 255 340 25 8 −68 2 6 12 12 2326 136 1813 1868274 2169 117 942 934137 54 16 787 5441 11 0 134 1247 23 118 434 3 5 0 142 1 −78 −100 −67 −54 8 12 9.9 12 1259 1452346 1116 1390789 84 4815 69 2227 67 3 62 2 −7 −52 11 7 9 9 254 5777 272 6895 50 106 31 155 197 18 114 22 −42 23 *Our calculated summary statistic (presented in table 4) used the DerSimonian–Laird method (DL) to pool these data, where each study’s DL weight was multiplied by the unweighted percentage difference shown. The DL effect sizes are not included here, as they are not scaled in a meaningful way. NICU, neonatal intensive care unit; PICU, pediatric intensive care unit. error types reported during the study’s pre-period were reduced by 65%, of the 164 errors reported during the study’s post-CPOE implementation period, 117 were related to the use of the CPOE system itself.19 The authors observed a range of CPOE-related errors, including prescribing errors, where clinicians chose the wrong drug or dose from a pull-down menu, attributed a prescription to the wrong patient, or entered duplicate orders, and transcribing errors, where pharmacy staff incorrectly recorded medication order information. As noted above, data from the Bradley et al study were not used in our meta-analysis, because the study used only a voluntarily reported error detection method. As a sensitivity analysis, we produced estimates including Bradley et al. Pooling data from Bradley et al with estimates from the other literature would decrease our estimate of 12.5% (bounds 10.6– 14.4%) error reduction nationally to 9.2% (bounds 0.33–36.7%), implying ∼3.1 million fewer averted medication errors per year. Reviewed studies used various medication error detection methods. Research suggests that the highest error rates are found through direct observation,21 followed by chart review, then automated surveillance, and voluntary reporting.22 There exists limited overlap in errors reported through each modality, and studies using a combination of methods detect the highest Table 4 error rates.23 We extracted information on detection mode from each study, and excluded studies relying solely on voluntary reporting or employing inconsistent detection modes before and after CPOE implementation. While there is a risk that errors will be double-counted in studies using multiple modes,22 studies that use single and/or less rigorous detection modes may seriously undercount medication errors. Definitions of medication errors also varied between studies in our review. For example, most studies counted wrong doses as medication errors, but only some counted drug–drug interactions. In addition, while some studies reported stratified results by error type (eg, frequency, route), others stratified by process stage (eg, prescribing, transcribing). Finally, some studies defined medication errors as events with potential to cause patient harm, while others counted errors corrected by the hospital’s internal redundant systems before reaching the patient. Unfortunately, in aggregating these data, it was not possible to parse out these differences. Several limitations should be noted when interpreting results. As with any meta-analysis, some limitations extend from limitations in the primary literature upon which our estimates relied. First, as noted above, detection mode and medication error Outcome and supporting statistics Calculated metric Supporting statistics Mean % reduction in medication error rates conditional on using CPOE to prescribe the order Proportion of medication orders that are ordered using a CPOE system Outcome statistics Percentage reduction in medication error frequency resulting from CPOE use to process medication orders Absolute reduction in medication error frequency resulting from CPOE use to process medication orders Data source Point estimate Estimate bound Peer-reviewed literature −48% (−55% to −41%) Hospital surveys (2007 AHA annual survey, 2008 EHR adoption database, 2006 ASHP National Survey) 26.10% (16.0% to 53.6%) Calculated from supporting statistics −12.5% (−14.4% to −10.6%) Calculated from supporting statistics 17390443 (88058 to 27094038) AHA, The American Hospital Association; ASHP, The American Society of Health-System Pharmacists; CPOE, computerized provider order entry; EHR, electronic health record. 474 Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241 Research and applications Figure 1 Estimated medication errors averted due to observed and increased use of computerized provider order entry in inpatient acute-care hospitals in a 1-year period. definitions varied across included studies. We attempted to address these and other sources of study heterogeneity by implementing inclusion criteria to ensure a minimum standard for comparability, and by using statistical adjustments on pooled estimates. However, it was not possible to fully account for all methodological differences. Second, hospitals in included studies may not be broadly representative of all US hospitals, particularly since there was a preponderance of large, urban, academic medical centers; Hug and colleagues found a higher incidence of ADEs in community hospitals than in academic medical centers.24 For this reason, the medication error rates and the mix of medication order types in included studies may vary from that in all US hospitals. Similarly, while studies have found that specific CPOE functionalities or accompanying clinical decision support can affect error rates,25 26 we could not systematically capture and quantify these factors for our estimate. It was difficult to observe and quantify other factors that may modify CPOE effects across hospitals (eg, implementation duration, commitment to quality improvement, severity of illness among patient population). Finally, we note an additional limitation not stemming from our meta-analytic approach. Our calculations assume that the effect of CPOE adoption on medication error rates is constant as the number of hospitals adopting CPOE increases. This assumption may not hold if later CPOE adopters differ systematically from earlier adopters. Our findings and the limitations of our estimate point to key areas for future research; as the relevant source literature grows in breadth and methodological consistency to overcome these limitations, the base methodology described here can be refined and expanded to improve on existing estimates. First, further research is needed to characterize the effect of CPOE implementation on order volume and patterns, and heterogeneity in the likelihood of medication errors for different types of medications, orders, and settings. In addition, common and consistently applied definitions of medication errors and serious medication errors, as well as consistent stratification of errors by type and/or ordering process stage, will ensure greater comparability across studies. Second, further work is needed to explain variation in findings across studies; for example, authors have noted an increase in medication errors due to CPOE19 and unintended consequences of health IT.27 28 Even among the eight studies in our meta-analysis showing reductions in errors due to CPOE, variation in the magnitude of impacts across study settings may indicate that not all patient populations will benefit equally from CPOE’s apparent error risk reduction. Third, given the variation in detected error rates by detection mode, future model refinements might weight analyses according to detectionmode sensitivity. Making scaling decisions will necessarily require a judgment about the relative value of more or less rigorous detection modes. Finally, additional evidence is needed to establish more concrete links between medication errors, ADEs, and patient harm; while Bates et al29 found that 0.9% of medication errors result in ADEs, few other such estimates are currently available, either in general or stratified by CPOE system functionality, and even less is known about the frequency with which medication errors result in actual patient harm. CONCLUSION Our rigorously developed meta-analytic estimate is in keeping with the earlier, evidence-based heuristic of Bates et al,23 and, more importantly, is a much-needed addition to our knowledge of the effect of CPOE on medication errors. Our estimation Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241 475 Research and applications approach may be useful for estimating other health IT-related impacts on the healthcare system and patient outcomes. Future research in this area will be critically important to inform policy and funding decisions regarding the development and implementation of CPOE in care delivery. 7 Acknowledgements We thank our Task Order Officer, Robert Mayes (Agency for Healthcare Research and Quality), who helped conceptualize this paper. Ashish Jha (Department of Health Policy and Management, Harvard School of Public Health) provided expertise on CPOE and medication errors and contributed to the literature review. Catherine Desroches and Eric Campbell (Institute for Health Policy, Massachusetts General Hospital) provided expertise on CPOE and data management for the American Hospital Association’s 2008 EHR Adoption Database. William Rhodes and K P Srinath (Abt Associates) contributed to the methodology to develop estimate bounds. We thank Jacob Klerman and the Abt Associates Journal Authors Support Group for financial support to revise and resubmit this manuscript. Finally, Rena Kirsch and Suzanne Erfurth (Abt Associates) provided editorial assistance. 9 Funding This project was funded under PSC contract No 233–02–0088, Task Order # HHSP233200700008T from the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services. The opinions expressed in this document are those of the authors and do not reflect the official position of AHRQ or the US Department of Health and Human Services. 8 10 11 12 13 14 15 Competing interests All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (uploaded through manuscript central) and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work. 16 Contributors MS and MW designed the study in response to the AHRQ RFP (the idea for this paper). MW developed the literature review method, with input from DR. MW drafted the original paper with DR and LO. DR led the analysis with substantial input from LO. SS and MS reviewed the original paper. SS and BB revised the paper for submission. SS, LO and DR responded to peer reviewers’ comments. 18 Provenance and peer review Not commissioned; externally peer reviewed. Open Access This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/ licenses/by-nc/3.0/ 17 19 20 21 22 23 REFERENCES 1 2 3 4 5 6 476 Aspden P, Wolcott J, Bootman J, et al. Preventing medication errors. Washington, DC: National Academic Press, 2007. Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA 1997;277:307–11. Lesar TS, Briceland LL, Delcoure K, et al. Medication prescribing errors in a teaching hospital. JAMA 1990;263:2329–34. Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: Institute of Medicine, National Academy Press, 2001. Berger RC, Kichak BA. Computerized physician order entry: helpful or harmful? J Am Med Inform Assoc 2004;11:100–3. Ammenwerth E, Schnell-Inderst P, Machan C, et al. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008;15:585–600. 24 25 26 27 28 29 Department of Health and Human Services, Centers for Medicare and Medicaid Services. 42 CFR parts 412, 413, 422 et al. Medicare and Medicaid Programs: Electronic Health Record Incentive Program; Final Rule. 2010. http://edocket.access. gpo.gov/2010/pdf/2010-17207.pdf (accessed 19 Aug 2010). Pedersen CA, Schneider PJ, Scheckelhoff DJ. ASHP national survey of pharmacy practice in hospital settings: monitoring and patient education–2006. Am J Health Syst Pharm 2007;64:507–20. Jha AK, DesRoches CM, Campbell EG, et al. Use of electronic health records in U.S. hospitals. N Engl J Med 2009;360:1628–38. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6:313–21. Taylor JA, Loan LA, Kamara J, et al. Medication administration variances before and after implementation of computerized physician order entry in a neonatal intensive care unit. Pediatrics 2008;121:123–8. Bizovi KE, Beckley BE, McDade MC, et al. The effect of computer-assisted prescription writing on emergency department prescription errors. Acad Emerg Med 2002;9:1168–75. Cordero L, Kuehn L, Kumar RR, et al. Impact of computerized physician order entry on clinical practice in a newborn intensive care unit. J Perinatol 2004;24:88–93. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338:232–8. Igboechi C, Ng C, Yang C, et al. Impact of computerized prescriber order entry on medication errors at an acute tertiary care hospital. Hosp Pharm 2003;38:227–31. Kim GR, Chen AR, Arceci RJ, et al. Error reduction in pediatric chemotherapy: computerized order entry and failure modes and effects analysis. Arch Pediatr Adolesc Med 2006;160:495–8. Mahoney CD, Berard-Collins CM, Coleman R, et al. Effects of an integrated clinical information system on medication safety in a multi-hospital setting. Am J Health Syst Pharm 2007;64:1969–77. Walsh KE, Landrigan CP, Adams WG, et al. Effect of computer order entry on prevention of serious medication errors in hospitalized children. Pediatrics 2008;121:e421–7. Bradley VM, Steltenkamp CL, Hite KB. Evaluation of reported medication errors before and after implementation of computerized practitioner order entry. J Health Inf Manag 2006;20:46–53. Egger M, Smith G, Altman D. Systematic reviews in health care: meta-analysis in context. 2nd edn. London: BMJ Publishing Group, 2001. Flynn EA, Barker KN, Pepper GA, et al. Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health Syst Pharm 2002;59:436–46. Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998;5:305–14. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280:1311–16. Hug BL, Witkowski DJ, Sox CM, et al. Adverse drug event rates in six community hospitals and the potential impact of computerized physician order entry for prevention. J Gen Intern Med 2009;25:31–8. Metzger J, Welebob E, Bates DW, et al. Mixed results in the safety performance of computerized physician order entry. Health Aff (Millwood) 2010;29:655–63. Nanji KC, Rothschild JM, Salzberg C, et al. Errors associated with outpatient computerized prescribing systems. J Am Med Inform Assoc 2011;18:767–73. Sittig DF, Ash JS, Guappone KP, et al. Assessing the anticipated consequences of computer-based provider order entry at three community hospitals using an open-ended, semi-structured survey instrument. Int J Med Inform 2008;77:440–7. Ash JS, Sittig DF, Dykstra R, et al. Exploring the unintended consequences of computerized physician order entry. Stud Health Technol Inform 2007;129(Pt 1):198–202. Bates DW, Boyle DL, Vander Vliet MB, et al. Relationship between medication errors and adverse drug events. J Gen Intern Med 1995;10:199–205. Radley DC, et al. J Am Med Inform Assoc 2013;20:470–476. doi:10.1136/amiajnl-2012-001241