Abstract
Background Acute kidney injury (AKI) is a common complication after cardiac surgery and is associated with worse outcomes. Its management relies on early diagnosis, and therefore, electronic alerts have been used to alert clinicians for development of AKI. Electronic alerts are, however, associated with high rates of alert fatigue.
Objectives We designed this study to assess the acceptance of user-centered electronic AKI alert by clinicians.
Methods We developed a user-centered electronic AKI alert that alerted clinicians of development of AKI in a persistent yet noninterruptive fashion. As the goal of the alert was to alert toward new or worsening AKI, it disappeared 48 hours after being activated. We assessed the acceptance of the alert using surveys at 6 and 12 months after the alert went live.
Results At 6 months after their implementation, 38.9% providers reported that they would not have recognized AKI as early as they did without this alert. This number increased to 66.7% by 12 months of survey. Most providers also shared that they re-dosed or discontinued medications earlier, provided earlier management of volume status, avoided intravenous contrast use, and evaluated patients by using point-of-care ultrasounds more due to the alert. Overall, 83.3% respondents reported satisfaction with the electronic AKI alerts at 6 months and 94.4% at 12 months.
Conclusion This study showed high rates of acceptance of a user-centered electronic AKI alert over time by clinicians taking care of patients with AKI.
Keywords: alert, clinical decision support, alert fatigue, electronic health record, user-centered alert
Background and Significance
Acute kidney injury (AKI), defined as an acute decline in kidney function, is one of the most common complications seen in critically ill patients. 1 It is especially common in patients after cardiac surgery 2 3 and is associated with a much higher risk of mortality 4 and development of chronic kidney disease. 5 The management of AKI relies on early diagnosis and preventing its progression by providing clinical interventions early in its course. 6 The diagnosis of AKI is, however, frequently missed or delayed among hospitalized patients. 7 There is, therefore, considerable interest in improving early recognition of AKI.
The increasing adoption of electronic health records (EHRs) over last two decades has provided a unique opportunity for EHRs to be used to improve early recognition of AKI by use of electronic AKI alerts. 8 This has been facilitated by the American Recovery and Reinvestment Act, Health Information Technology for Economic and Clinical Health Act, and incentive programs for the Centers for Medicare and Medicaid services aimed at increasing meaningful use of EHR in the United States. The last decade has thus seen increasing use of electronic AKI alerts in EHRs.
Although electronic alerts have been shown to improve patient outcomes, up to 96% electronic alerts are overridden and ignored. 9 Alert fatigue is a major contributor to the inappropriate dismissal and ineffectiveness of electronic alerts. 9 10 11 12 13 14 Recent literature has suggested alert fatigue to be a result of cognitive overload due to uninformative alerts and to desensitization to alerts over time. 15 Additionally, the interruptive nature of alerts, including specifically their location and timing within the workflow process, may itself lead to alert fatigue. 16 Thus, in addition to the type of information being shown, the manner in which this information is presented is also critical to avoid alert fatigue and increase the acceptance of alerts.
Objectives
We developed and implemented a noninterruptive, electronic AKI alert with a minimalistic design for the cardiac surgery intensive care unit (ICU) of a tertiary care, high volume cardiac surgery center starting July 2020. The alert was designed with an emphasis on a user-centered approach. The objective of this study was to assess the acceptance of this user-centered electronic AKI alert by clinicians and if it changes over time.
Methods
The Electronic AKI Alert
We developed this electronic AKI alert in the EPIC EHR. 17 It was developed in collaboration with physicians (A.S.: intensivist, nephrologist, and informaticist; J.P.: hospitalist and informaticist), informatics analyst at West Virginia University (WVU; R.L.N.), and analysts at EPIC Systems. We used previously validated logic to develop this electronic AKI alert. 18 19 Briefly, the median creatinine for the last 12 months was used to identify baseline creatinine for patients who had creatinine values available in our system. In the absence of prior creatinine values, we estimated the baseline based on back-calculation using the Modification of Diet in Renal Disease equation. In case of a history of chronic kidney disease and no documented prior creatinine values, admission creatinine was used as the baseline creatinine. The logic then compared the first creatinine from the current admission to that of the baseline creatinine and fired if an increase of 50% or more was detected. An increase of at least 0.3 mg/dL within 52 hours, and from baseline creatinine, was needed to fire the alert for subsequent creatinine values during that admission. A.S., J.P., and R.L.N. are members of the WVU Clinical Decision Support Committee. Due to the nonemergent nature of the information that needs to be conveyed by this alert, we chose it to be noninterruptive to minimize any disruptions in end-user workflow. We displayed the alert in the EPIC storyboard, which is a vertical bar situated on the left side of patient's chart and includes patient information such as name, photo, age, date of birth, allergies, etc. ( Fig. 1A ). This allowed for the alert to be easily visible throughout the end-users' interaction with the chart and without impeding the workflow. As shown in Fig. 1A , the alert displays as a “possible AKI” along with AKI stage. The alert calculates the stage by creatinine-based Kidney Disease: Improving Global Outcomes (KDIGO) AKI criteria. 6 This minimalistic design allows users to always see the high yield information. The users can then hover over or click on the alert to get detailed information regarding why the alert was fired as shown in Fig. 1B . Finally, as the goal of this alert was to alert clinicians toward new or worsening AKI, the electronic alert was programmed to disappear at 48 hours unless the continued rise in creatinine caused the alert to activate again. This electronic alert went live in our center's EPIC EHR on July 1, 2020, and was aimed at patients admitted in the cardiac surgery ICU at our tertiary care center.
Fig. 1.
The electronic AKI alert. ( A ) The EPIC Storyboard and the electronic AKI alert in it. ( B ) The detailed information that can be viewed by clicking or hovering over the electronic AKI alert. AKI, acute kidney injury.
Study Population and Survey
The study subjects were intensivists and advanced practice providers (APPs) in the cardiac surgery ICU at a large-volume cardiac surgery program. We utilized a previously validated survey to assess acceptance of electronic AKI alerts by the study subjects after institution of these alerts in EHR. 20 The survey consisted of questions to collect basic demographic information, questions about the alert, and questions to understand the providers' acceptance of these alerts. The latter questions were measured on a 5-point Likert scale (Strongly disagree, Disagree, Neither Disagree nor Agree, Agree, Strongly Agree) except for the final question which was an open-ended question asking if providers would recommend any changes to the electronic alerts. We administered the survey twice at 6 and 12 months and used the same survey for both rounds using REDCap. Subjects who had not completed the survey after 2 weeks received a reminder email for each timepoint.
Statistical Analysis
Our primary outcome was acceptance of electronic AKI alerts at 6 and 12 months. We considered the responses to the question, “In general, I am satisfied with the electronic AKI alerts” as the primary measure of acceptance of the electronic AKI alert at each time period. The responses “Agree” and “Strongly Agree” were considered as positive acceptance and other responses as lack of acceptance of the alert. We compared the responses between intensivists and APPs at each time period using chi-squared tests, and the correlation between various survey responses and the acceptance of the AKI alert using Spearman's correlation using RStudio (version 1.3.1093).
Results
Of the 19 providers we approached in the ICU (7 intensivists and 12 APPs), 18 (94.7%) responded to the survey at each of the two time periods. One APP did not respond to the survey during the 6-month time period and one intensivist did not respond during the 12-month time period.
At 6 months after their implementation, 38.9% providers reported that they recognized AKI sooner since the alert went live. This number increased to 66.7% by 12 months of survey. Most providers also shared that they re-dosed or discontinued medications earlier due to the AKI alert (88.2% at 6 months and 88.9% at 12 months). A majority also reported earlier modifications in management of volume status (72.2% at 6 months and 83.3% at 12 months), avoidance of intravenous (IV) contrast use (72.2% at 6 months and 83.3% at 12 months), and evaluation of patients by using point-of-care ultrasounds (POCUS; 77.8% at 6 months and 83.3% at 12 months) in response to the alert.
Acceptance of AKI Alert at 6 Month Survey
All providers received alerts in the 6-month period before this survey with majority of providers (63.3% APPs and all intensivists) reporting to have received two or less alerts a day. Overall, 15/18 (83.3%) respondents reported satisfaction with the electronic AKI alerts. There was no statistically significant difference in the satisfaction with alert between intensivists and APPs (5/7 intensivists for 71.4% vs. 10/11 APPs for 90.9% reported satisfaction, p = 0.3). Majority of providers (83.3%) reported satisfaction with the way this electronic alert is displayed in our EHR and 88.9% providers reported satisfaction with the duration the alert is active ( Table 1 ).
Table 1. Responses to survey at 6 months.
| Survey question | Response | Intensivist (7) n (%) |
APP (11) n (%) |
p -Value |
|---|---|---|---|---|
| The number of alerts I receive had a negative impact on overall patient care | Disagree/strongly disagree | 6 (85.7) | 10 (90.9) | 0.4 |
| Neither agree nor disagree | 1 (14.3) | 1 (9.1) | ||
| Agree/strongly agree | 0 (0.0) | 0 (0.0) | ||
| The number of alerts I received hindered my workflow | Never/rarely | 7 (100.0) | 11 (100.0) | 0.6 |
| Sometimes | 0 (0) | 0 (0) | ||
| Most of the time/all of the time | 0 (0) | 0 (0) | ||
| I was already aware that the patients had AKI prior to electronic alert being displayed | Never/rarely | 0 (0) | 0 (0) | 0.3 |
| Sometimes | 3 (42.9) | 4 (36.4) | ||
| Most of the time/all of the time | 4 (57.1) | 7 (63.6) | ||
| I would have recognized AKI regardless of the electronic alert |
Never/rarely | 0 (0) | 0 (0) | 0.3 |
| Sometimes | 2 (28.6) | 6 (54.6) | ||
| Most of the time/all of the time | 5 (71.4) | 5 (45.4) | ||
| I would not have recognized AKI as early as I did had it not been for the electronic alert | Disagree/strongly disagree | 3 (42.8) | 0 (0.0) | 0.05 |
| Neither agree nor disagree | 2 (28.6) | 6 (54.6) | ||
| Agree/strongly agree | 2 (28.6) | 5 (45.4) | ||
| In general, I have been documenting AKI (writing in Never the chart) as a diagnosis more frequently due to the electronic AKI alert system |
Never/rarely | 2 (28.6) | 3 (27.3) | 0.8 |
| Sometimes | 2 (28.6) | 5 (45.4) | ||
| Most of the time/all of the time | 3 (42.8) | 3 (27.3) | ||
| In general, me or my team have been led to re-dose or discontinue medications earlier due to the alerts from the AKI alert system a | Never/rarely | 1 (14.3) | 1 (9.1) | 0.6 |
| Sometimes | 3 (42.9) | 6 (54.5) | ||
| Most of the time/all of the time | 2 (28.6) | 4 (36.4) | ||
| In general, me or my team have been led to recommend re-dosing or discontinuing certain medications due to alerts from the AKI alert system | Never/rarely | 2 (28.6) | 3 (27.2) | 0.4 |
| Sometimes | 3 (42.8) | 4 (36.4) | ||
| Most of the time/all of the time | 2 (28.6) | 4 (36.4) | ||
| In general, me or my team have been led to change volume management early due to alerts from the AKI alert system |
Never/rarely | 3 (42.8) | 3 (27.2) | 0.2 |
| Sometimes | 2 (28.6) | 5 (45.5) | ||
| Most of the time/all of the time | 2 (28.6) | 4 (36.3) | ||
| In general, me or my team have been led to avoid testing with contrast due to alerts from the AKI alert system | Never/rarely | 4 (57.1) | 1 (9.1) | 0.1 |
| Sometimes | 2 (28.6) | 4 (36.4) | ||
| Most of the time/all of the time | 1 (14.3) | 6 (54.5) | ||
| In general, me or my team have been led to order urinalysis, urine electrolytes, and/or creatinine earlier due to the alerts from the AKI alert system |
Never/rarely | 2 (28.6) | 5 (45.5) | 0.4 |
| Sometimes | 4 (57.1) | 5 (45.5) | ||
| Most of the time/all of the time | 1 (14.3) | 1 (9.1) | ||
| In general, me or my team have been led to perform diagnostic point of care ultrasound (POCUS—including heart, lungs, kidneys, and urinary bladder) earlier due to alerts from the AKI alert system |
Never/rarely | 3 (42.8) | 1 (9.1) | 0.4 |
| Sometimes | 3 (42.8) | 5 (45.5) | ||
| Most of the time/all of the time | 1 (14.2) | 5 (45.5) | ||
| In general, me or my team have been led to review patient's inputs and outputs more closely due to alerts from the AKI alert system | Never/rarely | 3 (42.8) | 0 (0.0) | 0.1 |
| Sometimes | 2 (28.6) | 4 (36.4) | ||
| Most of the time/all of the time | 2 (28.6) | 7 (63.6) | ||
| In general, the care of my patients is overall improved due to the AKI alert system | Disagree/strongly disagree | 0 (0.0) | 1 (9.1) | 0.5 |
| Neither agree nor disagree | 4 (57.1) | 3 (27.3) | ||
| Agree/strongly agree | 3 (42.8) | 7 (63.6) | ||
| I am satisfied with the way the electronic AKI alerts are displayed | Disagree/strongly disagree | 0 (0.0) | 1 (9.1) | 0.2 |
| Neither agree nor disagree | 2 (28.6) | 0 (0.0) | ||
| Agree/strongly agree | 5 (71.4) | 10 (90.9) | ||
| I am satisfied with the duration the electronic AKI alerts are displayed | Disagree/strongly disagree | 0 (0.0) | 0 (0.0) | 0.08 |
| Neither agree nor disagree | 2 (28.6) | 0 (0.0) | ||
| Agree/strongly agree | 5 (71.4) | 11 (100.0) | ||
| In general, I am satisfied with the electronic AKI alerts | Disagree/strongly disagree | 0 (0.0) | 0 (0.0) | 0.1 |
| Neither agree nor disagree | 2 (28.6) | 1 (9.1) | ||
| Agree/strongly agree | 5 (71.4) | 10 (90.9) |
Only six intensivists answered this question.
The responses that correlated most (positively or negatively) with increased acceptance of the electronic alert in the 6-month survey were, “I am satisfied with the duration the electronic AKI alerts are displayed for” (Spearman rho: 0.85, p < 0.001), “I am satisfied with the way the electronic AKI alerts are displayed” (Spearman rho: 0.85, p < 0.001), “In general, the care of my patients is overall improved due to the AKI alert system” (Spearman rho: 0.56, p = 0.02), “In general, me or my team have been led to review patient's inputs and outputs more closely due to alerts from the AKI alert system” (Spearman rho: 0.50, p = 0.03), “In general, me or my team have been led to perform diagnostic POCUS (including heart, lungs, kidneys, and urinary bladder) earlier due to the alerts from the AKI alert system” (Spearman rho: 0.59, p = 0.009), “In general, me or my team have been led to avoid testing with contrast due to alerts from the AKI alert system” (Spearman rho: 0.55, p = 0.02), “The number of alerts I received hindered my workflow” (Spearman rho: −0.47, p = 0.04), and “The number of alerts I received had a negative impact on overall patient care” (Spearman rho: −0.61, p = 0.006).
Acceptance of AKI Alert at 12-Month Survey
At 12 months, 55.6% of providers reported receiving an average of one to two alerts or less per day and a total of 77.8% providers reported receiving an average of two to three alerts or less per day. Only one provider reported receiving more than five alerts daily. Overall, 17/18 (94.4%) of respondents reported satisfaction with the electronic AKI alerts. There was no statistically significant difference in the satisfaction with the alert between intensivists and APPs (6/6 intensivists for 100% vs. 11/12 APPs for 91.7% reported satisfaction, p = 0.2). Again, a vast majority of providers (94.4%) providers reported satisfaction with the way this electronic alert is displayed in our EHR and an equal number of providers reported satisfaction with the duration the alert is active for ( Table 2 ).
Table 2. Responses to survey at 12 months.
| Survey question | Response | Intensivist (6) n (%) |
APP (12) n (%) |
p -Value |
|---|---|---|---|---|
| The number of alerts I receive had a negative impact on overall patient care | Disagree/strongly disagree | 5 (83.3) | 10 (83.3) | 0.4 |
| Neither agree nor disagree | 1 (16.7) | 1 (8.3) | ||
| Agree/strongly agree | 0 | 1 (8.3) | ||
| The number of alerts I received hindered my workflow | Never/rarely | 6 (100) | 11 (91.7) | 0.6 |
| Sometimes | 0 (0) | 1 (8.3) | ||
| Most of the time/all of the time | 0 (0) | 0 (0) | ||
| I was already aware that the patients had AKI prior to electronic alert being displayed | Never/rarely | 0 (0) | 0 (0) | 0.2 |
| Sometimes | 2 (33.3) | 8 (66.7) | ||
| Most of the time/all of the time | 4 (66.7) | 4 (33.3) | ||
| I would have recognized AKI regardless of the electronic alert |
Never/rarely | 0 (0) | 0 (0) | 0.2 |
| Sometimes | 1 (16.7) | 5 (41.7) | ||
| Most of the time/all of the time | 5 (83.3) | 7 (58.3) | ||
| I would not have recognized AKI as early as I did had it not been for the electronic alert | Disagree/strongly disagree | 3 (50.0) | 0 (0.0) | 0.01 |
| Neither agree nor disagree | 0 (0.0) | 6 (50.0) | ||
| Agree/strongly agree | 3 (50.0) | 6 (50.0) | ||
| In general, I have been documenting AKI (writing in Never the chart) as a diagnosis more frequently due to the electronic AKI alert system |
Never/rarely | 1 (16.7) | 0 (0.0) | 0.4 |
| Sometimes | 3 (50.0) | 6 (50.0) | ||
| Most of the time/all of the time | 2 (33.3) | 6 (50.0) | ||
| In general, me or my team have been led to re-dose or discontinue medications earlier due to the alerts from the AKI alert system | Never/rarely | 2 (33.3) | 0 (0.0) | 0.1 |
| Sometimes | 3 (50.0) | 5 (41.7) | ||
| Most of the time/all of the time | 1 (16.7) | 7 (58.3) | ||
| In general, me or my team have been led to recommend re-dosing or discontinuing certain medications due to alerts from the AKI alert system | Never/rarely | 2 (33.3) | 0 (0.0) | 0.1 |
| Sometimes | 3 (50.0) | 5 (41.7) | ||
| Most of the time/all of the time | 1 (16.7) | 7 (58.3) | ||
| In general, me or my team have been led to change volume management early due to alerts from the AKI alert system |
Never/rarely | 2 (33.3) | 1 (8.3) | 0.2 |
| Sometimes | 3 (50.0) | 4 (33.3) | ||
| Most of the time/all of the time | 1 (16.7) | 7 (58.3) | ||
| In general, me or my team have been led to avoid testing with contrast due to alerts from the AKI alert system | Never/rarely | 2 (33.3) | 1 (8.3) | 0.2 |
| Sometimes | 2 (33.3) | 4 (33.3) | ||
| Most of the time/all of the time | 2 (33.3) | 7 (58.3) | ||
| In general, me or my team have been led to order urinalysis, urine electrolytes, and/or creatinine earlier due to the alerts from the AKI alert system |
Never/rarely | 4 (66.7) | 5 (41.7) | 0.2 |
| Sometimes | 1 (16.7) | 6 (50.0) | ||
| Most of the time/all of the time | 1 (16.7) | 1 (8.3) | ||
| In general, me or my team have been led to perform diagnostic point of care ultrasound (POCUS—including heart, lungs, kidneys, and urinary bladder) earlier due to alerts from the AKI alert system |
Never/rarely | 3 (50.0) | 0 (0.0) | 0.09 |
| Sometimes | 2 (33.3) | 5 (41.7) | ||
| Most of the time/all of the time | 1 (16.7) | 7 (58.3) | ||
| In general, me or my team have been led to review patient's inputs and outputs more closely due to alerts from the AKI alert system | Never/rarely | 3 (50.0) | 1 (8.3) | 0.1 |
| Sometimes | 1 (16.7) | 3 (25.0) | ||
| Most of the time/all of the time | 2 (33.3) | 8 (66.7) | ||
| In general, the care of my patients is overall improved due to the AKI alert system | Disagree/strongly disagree | 0 (0.0) | 0 (0.0) | 0.2 |
| Neither agree nor disagree | 2 (40.0) | 2 (16.7) | ||
| Agree/strongly agree | 3 (60.0) | 10 (83.3) | ||
| I am satisfied with the way the electronic AKI alerts are displayed | Disagree/strongly disagree | 0 (0.0) | 0 (0.0) | 0.2 |
| Neither agree nor disagree | 0 (0.0) | 1 (8.3) | ||
| Agree/strongly agree | 6 (100.0) | 11 (91.7) | ||
| I am satisfied with the duration the electronic AKI alerts are displayed | Disagree/strongly disagree | 0 (0.0) | 0 (0.0) | 0.2 |
| Neither agree nor disagree | 0 (0.0) | 1 (8.3) | ||
| Agree/strongly agree | 6 (100.0) | 11 (91.7) | ||
| In general, I am satisfied with the electronic AKI alerts | Disagree/strongly disagree | 0 (0.0) | 0 (0.0) | 0.2 |
| Neither agree nor disagree | 0 (0.0) | 1 (8.3) | ||
| Agree/strongly agree | 6 (100.0) | 11 (91.7) |
The responses that correlated the most with increased acceptance of the electronic alert in the 12-month survey were, “I am satisfied with the duration the electronic AKI alerts are displayed for” (Spearman's rho: 1.00, p < 0.001), “I am satisfied with the way the electronic AKI alerts are displayed” (Spearman's rho: 1.00, p < 0.001), “In general, me or my team have been led to recommend re-dosing or discontinuing certain medications due to alerts from the AKI alert system” (Spearman's rho: 0.59, p = 0.009), and “The number of alerts I received hindered my workflow” (Spearman's rho: −0.49, p = 0.03).
Discussion
Our study showed that there was a high level of satisfaction with electronic AKI alerts among physicians and APPs at a large-volume cardiac surgery ICU 6 months following implementation. Satisfaction remained high at 12 months after the implementation of the AKI alerts.
This electronic AKI alert was developed by a team of informatics analysts and physician-informaticists with clinical training and experience in nephrology, critical care medicine, and hospital medicine. The emphasis from the initial stage of development of this alert was to ensure a user-centered design. Such an approach that focuses on the users' goals, motivations, and environment throughout the process of design 21 has been shown to increase the adoption of health information technology tools. 22 23
Previous literature has shown that uninformative alerts are associated with a high degree of alert fatigue. 15 In our study the majority of providers reported that they recognized AKI sooner after the alert went live. They also reported that this alert led them to re-dose/discontinue medications, avoid IV contrast use, make earlier modifications in management of volume status, and evaluate patients with point-of-care ultrasounds. As these are all important elements of management for patients with AKI, it shows the clinical relevance of the alert as perceived by the providers.
As the definition of AKI has evolved over time, 6 24 we used the latest and widely accepted KDIGO definition and utilized previously validated logic 18 19 to create this alert. In addition, the interruptive nature of alerts can also contribute to alert fatigue. This was shown in the study by Elias and colleagues 16 where they found that majority of interruptive alerts were closed by end-users in under 3 seconds. This raises the concern that most alerts are closed by providers even before they have had a chance to read them, thus rendering them ineffective. Once interruptive alerts are closed by the users, they are silenced until next time the chart is opened or the conditions for the alert to fire are met. Thus, this rapid dismissal of interruptive alerts makes them ineffective for clinical care for that entire time period. Our team therefore chose noninterruptive style of alerts to minimize any disruptions of the workflow of end users. As shown in Fig. 1A , we displayed this alert in the EPIC storyboard where it had an important strategic advantage of being always visible , regardless of the section of the chart the end user is in. We were thus able to provide the information about AKI to clinicians at all times, without interrupting their workflow. To further minimize the time and effort users have to spend reviewing this alert, we chose to display only whether someone may have AKI with its stage in the EPIC storyboard. The goal behind this minimalistic design was to ensure that users can always see the highest yield information. We did have the functionality where users could hover or click over the alert to get detailed information regarding why the alert was fired as shown in Fig. 1B . This design was well accepted by end users with 17/18 (94.4%) of providers responding that they were satisfied with the way the alerts were displayed 12 months out of their implementation.
As AKI is seen in over one-third patients undergoing cardiac surgery, it was important to make sure the alert deactivates on its own after alerting clinicians of AKI. Without this functionality the presence of alert in EHR risked becoming redundant and uninformative. As persistence of AKI beyond 48 hours is associated with worse outcomes in patients after cardiac surgery, 25 we chose this 48-hour period as the actionable window to keep the alert active. This ensured that the presence of the alert in the chart meant either new or worsening AKI in last 48 hours, thus making it clinically relevant and informative for clinicians taking care of these patients. At 12 months since the implementation of these alerts, 17/18 (94.4%) of providers responded that they were satisfied with this duration of display of alerts in the EHR.
Our study shows that it is possible to develop user-centered electronic alerts which are well accepted by end users. It is, however, important to acknowledge several limitations of this study. This study was conducted in a single cardiac surgery ICU of a large-volume cardiac surgery center. The results from this study may thus not be generalizable to other clinical settings or centers. Our alert was developed in EPIC Systems EHR. Although EPIC Systems is one of the widely utilized EHRs in the United States, 26 these designs and results may not be directly applicable to centers that use other EHRs. We do, however, believe that regardless of the EHR vendor the principles of user-centered design for developing alerts stay the same and this study convincingly shows that the user-centered design is an important aspect to develop patient care alerts. Although this study showed that the noninterruptive, electronic AKI alert was well accepted by end users, it does not provide a comparison against interruptive alerts. The interruptive alerts, however, are known to disrupt workflow and are associated with alert fatigue. 16 It is also not possible to tease out if one of the design features (information provided, noninterruptive nature, placement in EPIC storyboard, minimalistic design that could be expanded by hovering over or clicking on the alert, disappearance after 48 hours) contributed more to the acceptance of the alert by the end user than the other. Even though the providers participating in our surveys were the same at both the time periods, the surveys were themselves de-identified and unlinked. We were therefore not able to directly compare the responses of individuals between the two surveys. It is still important to note that between surveys administered at 6 and 12 months, alert acceptance increased from 15/18 (83.3%) to 17/18 (94.4%). This result contrasts with previous work on AKI alert acceptance, which showed alerts were generally well received but acceptance waned with time. 20 A risk of recall bias is also an inherent limitation of the study design. Finally, it is important to note that the focus of this study was to assess the acceptance of the electronic AKI alerts. It, therefore, does not investigate a different facet of clinical decision support system research—whether this electronic alert leads to actual change in clinical management and outcomes for patients.
Conclusion
In conclusion, this study demonstrates high rates of acceptance of a user-centered electronic AKI alert over time by clinicians taking care of patients at high risk for AKI and its complications. The results of our study highlight the importance of incorporating user-centered designs when developing alerts and underscore the need for further larger, multicenter studies in this area.
Clinical Relevance Statement
Electronic alerts are an important tool which when utilized effectively can greatly improve patient care. They, however, suffer from the risk of causing alert fatigue which greatly limits their usefulness. We have shown in this study that by focusing on developing a user-centered alert, it is possible to achieve high levels of user acceptance with electronic alerts which is sustained over time. This work, thus, addresses a highly clinically relevant issue and proposes a simple solution while harnessing the power of informatics.
Multiple-Choice Questions
-
Which of the following features in electronic alerts are associated with lower risk for development of alert fatigue?
Uninformative alert
Interruptive alert
Noninterruptive alerts
Too many alerts
Correct Answer: The correct answer is option c. Uninformative alerts and too many alerts can both lead to cognitive overload ultimately leading to development of alert fatigue. Interruptive nature of alerts is itself associated with development of alert fatigue.
-
Where in EPIC EHR was the electronic AKI alert displayed?
Results Review tab
Chart Review tab
Storyboard
Interruptive alert that displayed every time the chart was accessed
Correct Answer: The correct answer is option c. The electronic AKI alert was displayed in EPIC storyboard where it had an important strategic advantage of being always visible, regardless of the section of the chart the end-user is in. We were thus able to provide the information about AKI to clinicians at all times, without interrupting their workflow.
Funding Statement
Funding The project described was supported by the National Institute of General Medical Services (U.S. Department of Health and Human Services, National Institutes of Health, National Institute of General Medical Sciences, grant no.: 5U54GM104942-03) and National Institute of Diabetes and Digestive and Kidney Diseases (U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases, grant no.: 1K08DK131286-01A1). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Conflict of Interest A.S. has been supported by the WVCTSI Research Scholar Award through the National Institute of General Medical Services (U.S. Department of Health and Human Services, National Institutes of Health, National Institute of General Medical Sciences, grant no.: 5U54GM104942-03) and is currently supported by the National Institute of Diabetes and Digestive and Kidney Diseases (U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases, grant no.: 1K08DK131286-01A1).
Protection of Human and Animal Subjects
The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by WVU Institutional Review Board. Participants were informed of the contents prior to study participation and voluntarily consented to participate.
References
- 1.Milwaukee Initiative in Critical Care Outcomes Research (MICCOR) Group of Investigators . Kumar G, Kumar N, Taneja A. Nationwide trends of severe sepsis in the 21st century (2000-2007) Chest. 2011;140(05):1223–1231. doi: 10.1378/chest.11-0352. [DOI] [PubMed] [Google Scholar]
- 2.Priyanka P, Zarbock A, Izawa J, Gleason T G, Renfurm R W, Kellum J A. The impact of acute kidney injury by serum creatinine or urine output criteria on major adverse kidney events in cardiac surgery patients. J Thorac Cardiovasc Surg. 2021;162(01):143–1.51E9. doi: 10.1016/j.jtcvs.2019.11.137. [DOI] [PubMed] [Google Scholar]
- 3.Ramos K A, Dias C B. Acute kidney injury after cardiac surgery in patients without chronic kidney disease. Rev Bras Cir Cardiovasc. 2018;33(05):454–461. doi: 10.21470/1678-9741-2018-0084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Karkouti K, Wijeysundera D N, Yau T M. Acute kidney injury after cardiac surgery: focus on modifiable risk factors. Circulation. 2009;119(04):495–502. doi: 10.1161/CIRCULATIONAHA.108.786913. [DOI] [PubMed] [Google Scholar]
- 5.Ponte B, Felipe C, Muriel A, Tenorio M T, Liaño F. Long-term functional evolution after an acute kidney injury: a 10-year study. Nephrol Dial Transplant. 2008;23(12):3859–3866. doi: 10.1093/ndt/gfn398. [DOI] [PubMed] [Google Scholar]
- 6.Kellum J A, Lameire N, Aspelin P. Kidney disease: improving global outcomes (KDIGO) acute kidney injury work group. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Supplements. 2012;2(01):1–138. [Google Scholar]
- 7.Cheng X, Wu B, Liu Y, Mao H, Xing C. Incidence and diagnosis of acute kidney injury in hospitalized adult patients: a retrospective observational study in a tertiary teaching Hospital in Southeast China. BMC Nephrol. 2017;18(01):203. doi: 10.1186/s12882-017-0622-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Park S, Baek S H, Ahn S. Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: a quality improvement study. Am J Kidney Dis. 2018;71(01):9–19. doi: 10.1053/j.ajkd.2017.06.008. [DOI] [PubMed] [Google Scholar]
- 9.van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(02):138–147. doi: 10.1197/jamia.M1809. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Carspecken C W, Sharek P J, Longhurst C, Pageler N M. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics. 2013;131(06):e1970–e1973. doi: 10.1542/peds.2012-3252. [DOI] [PubMed] [Google Scholar]
- 11.Weingart S N, Toth M, Sands D Z, Aronson M D, Davis R B, Phillips R S. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163(21):2625–2631. doi: 10.1001/archinte.163.21.2625. [DOI] [PubMed] [Google Scholar]
- 12.Oppenheim M I, Vidal C, Velasco F T. Impact of a computerized alert during physician order entry on medication dosing in patients with renal impairment. Proc AMIA Symp. 2002:577–581. [PMC free article] [PubMed] [Google Scholar]
- 13.Nanji K C, Slight S P, Seger D L. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(03):487–491. doi: 10.1136/amiajnl-2013-001813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Phansalkar S, van der Sijs H, Tucker A D. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20(03):489–493. doi: 10.1136/amiajnl-2012-001089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.with the HITEC Investigators . Ancker J S, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(01):36. doi: 10.1186/s12911-017-0430-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar A M. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden. Appl Clin Inform. 2019;10(05):909–917. doi: 10.1055/s-0039-1700869. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.EPIC Accessed September 5, 2022 at:https://www.epic.com/software
- 18.Sakhuja A, Bataineh A, Dealmeida D. Creating a high-specificity acute kidney injury detection system for clinical and research applications. Am J Kidney Dis. 2021;78(05):764–766. doi: 10.1053/j.ajkd.2021.03.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Al-Jaghbeer M, Dealmeida D, Bilderback A, Ambrosino R, Kellum J A. Clinical decision support for in-hospital AKI. J Am Soc Nephrol. 2018;29(02):654–660. doi: 10.1681/ASN.2017070765. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Oh J, Bia J R, Ubaid-Ullah M, Testani J M, Wilson F P. Provider acceptance of an automated electronic alert for acute kidney injury. Clin Kidney J. 2016;9(04):567–571. doi: 10.1093/ckj/sfw054. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Del Beccaro M A, Jeffries H E, Eisenberg M A, Harry E D. Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit. Pediatrics. 2006;118(01):290–295. doi: 10.1542/peds.2006-0367. [DOI] [PubMed] [Google Scholar]
- 22.Luna D, Quispe M, Gonzalez Z. User-centered design to develop clinical applications. Literature review. Stud Health Technol Inform. 2015;216:967. [PubMed] [Google Scholar]
- 23.Russ A L, Chen S, Melton B L. A novel design for drug-drug interaction alerts improves prescribing efficiency. Jt Comm J Qual Patient Saf. 2015;41(09):396–405. doi: 10.1016/s1553-7250(15)41051-7. [DOI] [PubMed] [Google Scholar]
- 24.ANZICS Database Management Committe . Bagshaw S M, George C, Bellomo R. A comparison of the RIFLE and AKIN criteria for acute kidney injury in critically ill patients. Nephrol Dial Transplant. 2008;23(05):1569–1574. doi: 10.1093/ndt/gfn009. [DOI] [PubMed] [Google Scholar]
- 25.Brown J R, Kramer R S, Coca S G, Parikh C R. Duration of acute kidney injury impacts long-term survival after cardiac surgery. Ann Thorac Surg. 2010;90(04):1142–1148. doi: 10.1016/j.athoracsur.2010.04.039. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Koppel R, Lehmann C U. Implications of an emerging EHR monoculture for hospitals and healthcare systems. J Am Med Inform Assoc. 2015;22(02):465–471. doi: 10.1136/amiajnl-2014-003023. [DOI] [PubMed] [Google Scholar]