Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
PMC home page
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2023 Jul 11;30(10):1622–1633. doi: 10.1093/jamia/ocad129

Predicting emergency department visits and hospitalizations for patients with heart failure in home healthcare using a time series risk model

Sena Chae 1,✉,b, Anahita Davoudi 2,b, Jiyoun Song 3, Lauren Evans 4, Mollie Hobensack 5, Kathryn H Bowles 6,7, Margaret V McDonald 8, Yolanda Barrón 9, Sarah Collins Rossetti 10,11, Kenrick Cato 12, Sridevi Sridharan 13, Maxim Topaz 14,15,16
PMCID: PMC10531127  PMID: 37433577

Abstract

Objectives

Little is known about proactive risk assessment concerning emergency department (ED) visits and hospitalizations in patients with heart failure (HF) who receive home healthcare (HHC) services. This study developed a time series risk model for predicting ED visits and hospitalizations in patients with HF using longitudinal electronic health record data. We also explored which data sources yield the best-performing models over various time windows.

Materials and Methods

We used data collected from 9362 patients from a large HHC agency. We iteratively developed risk models using both structured (eg, standard assessment tools, vital signs, visit characteristics) and unstructured data (eg, clinical notes). Seven specific sets of variables included: (1) the Outcome and Assessment Information Set, (2) vital signs, (3) visit characteristics, (4) rule-based natural language processing-derived variables, (5) term frequency-inverse document frequency variables, (6) Bio-Clinical Bidirectional Encoder Representations from Transformers variables, and (7) topic modeling. Risk models were developed for 18 time windows (1–15, 30, 45, and 60 days) before an ED visit or hospitalization. Risk prediction performances were compared using recall, precision, accuracy, F1, and area under the receiver operating curve (AUC).

Results

The best-performing model was built using a combination of all 7 sets of variables and the time window of 4 days before an ED visit or hospitalization (AUC = 0.89 and F1 = 0.69).

Discussion and Conclusion

This prediction model suggests that HHC clinicians can identify patients with HF at risk for visiting the ED or hospitalization within 4 days before the event, allowing for earlier targeted interventions.

Keywords: heart failure, home care services, natural language processing, electronic health records, nursing informatics, machine learning

INTRODUCTION

Every year, more than 11 000 home healthcare (HHC) agencies across the United States (US) provide care to more than 3.4 million older adults.1 One in 3 HHC patients is diagnosed with heart failure2 (HF)—a chronic condition that causes high levels of symptom burden, which results in low quality of life.3,4 Despite efforts to improve care for patients with HF, negative outcomes remain prevalent.5 Specifically, hospitalization rates for patients with HF remain relatively high (∼17%).5 Treatment of HF has a direct cost of over $34 billion per year, with hospitalizations accounting for the majority of the costs.5 Furthermore, more than 1 million emergency department (ED) visits for HF per year highlights the severity of the condition and the need for early detection and proper management of HF.6

Patients with HF in HHC are frequently hospitalized for reasons related to symptom aggravation (eg, dyspnea, fluid overload) and comorbidity burden.7–9 Symptom presentation may occur days before negative outcomes, such as ED visits or hospitalizations; 1 study reported dyspnea presented on average 3 days before hospitalization.8 Hence, close monitoring of symptoms and timely intervention based on risk prediction may allow HHC clinicians (registered nurses, social workers, physical, and occupational therapists) to prevent ED visits or hospitalizations.10,11 HHC clinicians can also help at-risk patients to improve their self-management skills, leading to better outcomes.12 The most appropriate time to have early intervention with patients with HF in the HHC setting would depend on the specific architecture and experiment of the predictive model and the patients being monitored. Therefore, it would be ideal to design a predictive model using time-variant temporal variables to identify patients who are at risk and intervene as soon as possible to prevent negative outcomes. The exact timing would need to be determined by further testing and implementing the clinical decision support tools in the HHC setting.

However, no previous studies have demonstrated the feasibility of using time series risk prediction models based on routinely collected electronic health record (EHR) data to identify at-risk patients in HHC.13 A significant amount of information on potential risk factors, which is not always present in structured data (eg, standardized assessments, vital signs), is often recorded in clinical notes.14 Our group has previously developed and validated natural language processing (NLP) approaches to identify and extract “concerning” notes,15 as well as potential risk factors,15,16 symptoms,17 and poor self-management in patients with HF18 from HHC clinical notes.

However, these NLP algorithms have not yet been integrated into predictive modeling in a manner that allows us to take into consideration the dynamic visit-to-visit changes in patients’ health status. Several models predicting ED visits and hospitalizations have been developed; however, these models focus on the hospital setting and primarily use limited data (mostly administrative)16–20 for risk profiling. In addition, patients’ assessments are documented irregularly and are asynchronously extracted from HHC EHRs. Furthermore, patients may experience dynamic and nonlinear symptom severity or condition changes across treatment trajectories. Data issues, such as varying time gaps between record points, and fluctuating and nonlinear longitudinal symptom dynamics across the HF treatment continuum, make the analysis of EHR data challenging. While traditional statistical analysis using EHRs has not been successful in dealing with these challenges,19 this study explores the feasibility of comprehensive and time series risk modeling using longitudinal EHR data in the HHC setting for patients with HF.

This study is the first to integrate comprehensive patient information across the HHC EHR into a time series risk prediction models for ED visit and hospitalization risk. The specific aims of this study are: (1) to develop a time series risk model for predicting risk for ED visits and hospitalizations in patients with HF using longitudinal EHR data, (2) to determine what combined datasets of variables result in creating the best-performing risk models over various outcome time windows, and (3) to identify the highly correlated variables associated with increased risk for ED visits and hospitalizations.

MATERIALS AND METHODS

Study design and study population

We extracted data for all patients with HF (ICD-10 codes 50.x, I11.0, I13.0, I13.1, I13.2)20 admitted between January 1, 2015 and December 31, 2017 to one of the largest nonprofit HHC organizations in the Northeastern US. The unit of analysis was 1 HHC visit, defined as any visit provided by HHC health care providers (eg, registered nurses, physical therapists, and social workers). A HHC episode was defined as all services provided during the time between HHC admission and HHC discharge.

We used structured data (Outcome and Assessment Information Set [OASIS], vital signs, visit characteristics) and unstructured data (HHC clinical notes) with 7 different sets of variables explained in the “Variable Selection” section below in our analysis. We extracted all clinical notes (n = 125 979) including visit notes and care coordination notes generated by mostly nurses, as well as some physical and occupational therapists and social workers during or between HHC visits. Visit notes describe the care provided and the patient’s status during an HHC visit. Care coordination notes document communication between HHC health care providers (eg, calling a physician) and other care-related activities (eg, ordering wound care supplies).

Study outcome

Our primary outcome of interest was hospitalization anytime within a 60-day HHC episode or ED visits. This was determined in accordance to Medicare reimbursement for HHC for up to 60 days.21

Variable selection and the final dataset preparation

We created 7 sets of variables starting with: (1) the OASIS, and then adding (2) vital signs, (3) visit characteristics, (4) rule-based NLP algorithm, (5) term frequency-inverse document frequency (TF-IDF), (6) Bio-Clinical Bidirectional Encoder Representations from Transformers (BERT), and (7) topic modeling for the analysis. Each set of variables is described below.

Set of variables 1: OASIS data

In order to select variables to be incorporated into a risk prediction model, we used univariate analysis (ie, t tests for continuous variables and chi-square tests for categorical variables) at the statistically significant level with P < .01. From the start of care OASIS, sociodemographic characteristics were examined at the patient level, while clinical characteristics were examined at the episodic level to be compared with those with ED visits or hospitalizations, HF or other related reasons, without ED visits or hospitalizations. To select clinically meaningful variables, we consulted with 5 HHC and informatics experts (J.S., K.B., M.M., Y.B., and M.T.) who have extensive experience in HHC nursing, research, and cardiology. Based on this discussion, the selected variables were deemed conceptually associated with risk for ED visits and hospitalizations in patients with HF in HHC settings; therefore, they are included in the final data set for building a risk model. All OASIS variables selected are listed in Supplementary Table S1.

Set of variables 2: vital signs

Similar to other studies,22,23 we used 2 cardiovascular-related vital signs, blood pressure and pulse rate, since both are routinely monitored by HHC healthcare providers and are associated with hospitalization or ED visits for patients with HF. Blood pressure was scored as 0 = missing, 1 = normal (less than 120/80 mm Hg), 2 = elevated (systolic between 120 and 129 mm Hg and diastolic less than 80 mm Hg), 3 = stage 1 hypertension (systolic between 130 and 139 mm Hg or diastolic between 80 and 89 mm Hg), 4 = stage 2 hypertension (systolic at least 140 mm Hg or diastolic at least 90 mm Hg), or 5 = hypertensive crisis (systolic over 180 mm Hg and/or diastolic over 120 mm Hg), as per the 2017 American College of Cardiology/American Heart Association (ACC/AHA) guidelines.24 Blood pressure could be measured up to 3 times as needed during a visit. The total number of blood pressure measurements (a sum of first, second, and third blood pressure measurement scores, where 0 means no blood pressure measurement recorded at the visit, 1 means that 1 blood pressure reading was recorded, and 2 means 3 blood pressure readings) were recorded. Pulse rate was used as a continuous variable. The analysis used 3 variables—the ACC/AHA blood pressure classification, the number of blood pressure measurements taken, and the heart rate—to develop a set of variables.

Set of variables 3: visit characteristics

We extracted the following variables from administrative data, including time-series visits information and visit purpose to create visit characteristics: the number of days after admission (visit date—admission date); and the purpose of the visit (eg, rehabilitation-related, nursing training, and education). The term discharge includes: (1) discharge from HHC when HHC services are no longer required for patients without hospitalization or ED visits and (2) discharge from HHC due to acute care utilization for patients with hospitalization or ED visits (Supplementary Table S1).

Set of variables 4: NLP technique 1—rule-based NLP-derived variables

We applied 4 different NLP techniques using the same clinical notes because different NLP techniques provide different performance and interpretability.25 For creating the set of Variables 4, we used previously developed and validated NLP approaches to extract symptoms, “concerning” notes, and risk factors for hospitalization and ED visits from HHC clinical notes. The methods briefly summarized below are fully described in previous publications.15–18 We merged the following 2 episode-level data sets from preliminary work into a single visit-level data set: (1) HF data set: HF patient characteristics and symptoms documented in HHC clinical notes associated with ED visits and hospitalizations (n = 9362 HF patients who received 12 223 episodes); and (2) risk factor data set: potential risk factors and “concerning” notes for ED visits and hospitalizations extracted from clinical notes (n = 66 317 patients who received 86 866 HHC episodes). We used a total of 46 rule-based NLP-derived variables: 12 HF symptoms (eg, dyspnea, fatigue etc.), the total number of symptoms, and poor self-management,17,18 “concerning” notes and “having a problem,” and 30 general risk factors (Supplementary Table S1).

NLP approach no. 1: HF symptoms and poor self-management indicators

Based on relevant literature, a standardized health terminology (the Omaha System), and expert consensus, we identified 12 symptom domains relevant to HF in HHC (anorexia, chest pain, confusion, cough, dizziness, dyspnea, fatigue, nausea, palpitation, peripheral edema, weight loss, and weight gain).17,26 Next, we used an open-source NLP tool called NimbleMiner27 to expand and refine synonymous terms for each symptom domain. If a patient had at least 1 instance of a documented symptom, they were classified as having a symptom.

In addition, our team identified HHC patients with HF who have poor self-management by applying rule-based NLP to clinical notes.18 Six domains of HF self-management were identified: poor diet adherence, poor medication adherence, poor exercise/physical activity tolerance, issues with other self-care activities/self-monitoring, missed healthcare encounters, and unspecified nonadherence. If a patient had at least 1 instance of documented poor self-management, they were classified as having poor self-management. Our risk prediction model incorporated statistically significant (P < .01) and clinically meaningful variables from these previous analyses.14,18

NLP approach no. 2: “concerning” clinical notes

Previously, our team developed machine learning based NLP methods to classify HHC clinical notes as either “concerning” or “not concerning.”15 A “concerning” note was defined as a note including 1 or more risk factors associated with deterioration, thus resulting in ED visits or hospitalizations. We applied Convolutional Neural Networks (CNN), which demonstrated better performance for the binary classification task to classify each clinical note as either “concerning” or “not concerning.”16

NLP approach no. 3: general hospitalization and ED visit risk factors

General risk factors for ED visits and hospitalization during HHC visits were extracted from HHC clinical notes using a valid, rule-based NLP algorithm based on the Omaha System, a standardized nursing terminology.15,16 Omaha System problems such as “Circulation,” “Bowel function,” and “Abuse” were identified as high risk factors associated with ED visits or hospitalization during HHC. The methods for our dataset preparation are fully described in Supplementary Table S2.

Set of variables 5: NLP technique 2-term frequency-inverse document frequency (TF-IDF) and lexical features

We generated TF-IDF vectors for each clinical note to count the word weight by considering the term frequency (TF) and inverse document frequency (IDF). TF reflects the frequency of a term within a note, while IDF assigns higher weight to less frequent words.28 TF-IDF quantifies word relevance in a document, and this information can be used to build predictive models that can classify or analyze text data.29

Set of variables 6: NLP technique 3—pre-trained language model (Bio-Clinical BERT)

We used Bio-Clinical BERT, a pretrained NLP model that uses a large amount of health-related text on the web.30–32 A state-of-the-art neural language model, Bio-Clinical BERT, which is trained on large amounts of biomedical data, such as medical records and scientific articles, achieved the best performance in comparison with conventional machine learning models.33 Its ability to accurately process and extract information from large amounts of biomedical text data makes it a valuable tool for building a predictive model that can be trained to identify the presence of negative outcomes based on patient symptoms and medical history. In this study, we generated the Bio-Clinical BERT vectors for all the available clinical notes at each HHC visit.

Set of variables 7: NLP technique 4—topic modeling

We applied Latent Dirichlet Allocation (LDA) topic modeling to extract another variable set of the inherent latent topics of HHC clinical notes. LDA is a technique for content analysis designed to automatically organize large sets of documents based on latent topics, measured as patterns of word (co-)occurrence.34 The resulting topics can then be used as variables in a predictive model, providing additional information about the content of the documents that can be used to make predictions.35 We ran the model for 10, 20, and 30 topics, calculated the F1 score for each model, and selected the 10 topic models that demonstrated the highest F1 score. We generated the topic models for all the available clinical notes at each HHC visit.

Building risk prediction models

Machine learning model development and evaluation

To address Aim 1, we developed a risk prediction model using a machine learning approach with an open-source AutoGluon-Tabular classifier (version v 0.6.0).

Different risk models, such as the Cox Proportional Hazards model (CPH), have their own strengths and limitations. The AutoGluon-Tabular classifier prioritizes predictive performance, while the CPH model offers interpretability through hazard ratio estimation.36 However, the CPH model assumes the proportional hazards assumption, which may not hold true in our study due to the changing risk levels for ED visits or hospitalization over time. In this study, our goal is to identify the best-performing risk models for different outcome time windows. Given this objective, and the limitations of applying the CPH to this study, the AutoGluon-Tabular classifier aligns better with our research goals.

AutoGluon-Tabular automatically selects best-performing algorithms and hyperparameters tuning for effective application of machine learning.37 AutoGluon streamlines the machine learning pipeline by incorporating automated hyperparameter tuning, making it easier for users to achieve high-performing models without manual adjustments.38 This approach allows users to obtain optimized results without specifying hyperparameters or comprehending the optimization process in-depth. We developed 7 risk prediction models with additive datasets that build off of each other: (ie, 1. OASIS only, 2. OASIS + vital signs, … 7. OASIS + vital signs + visit characteristics + rule-based NLP-derived variables + TF-IDF+Bio-Clinical BERT + topic modeling etc.). The rationale for our data processing order is based on the accessibility of each data source: We initiated our analysis with OASIS, a universally available and federally required standardized assessment for all HHC agency patients. Following this, we integrated increasingly complex data sources, necessitating more extraction effort from raw HHC data. This process began with vital signs and HHC visit characteristics, and we proceeded to include NLP-extracted variables such as rule-based, BERT, TF-IDF, and topic models.

For Aim 2, we experimented with several risk prediction time windows as an outcome of the risk prediction model based on the time frame from previous research for predicting adverse events in patients with HF.39 Specifically, we developed 15 models to predict hospitalization or ED visit over each day within 2 weeks (1–15 days). Existing literature suggests that HHC patients are at heightened risk for hospitalizations and ED visits in the first 2 weeks of HHC services.40,41 Therefore, we wanted to examine the performance of risk prediction models at every day within those first 2 weeks of HHC services (1–15 days). Further, we also wanted to explore the longer-term performance of risk prediction models with intervals of 2 weeks, specifically at 30, 45, and 60 days. Our time period is limited to 60 days because HHC episodes are mostly limited to 60 days by the payer (Center for Medicare and Medicaid Services). For each time window, we collected the most recent values from model variables generated at least that many days before the next ED visit or hospitalization. For example, when the time window was 7 days, the duration of the time we are predicting the events (ED or hospitalization) is, at the most, 7 days (1–7 days).

The dataset included 25 OASIS variables, 3 vital signs, 2 visit characteristics, 46 rule-based NLP-derived variables, TF-IDF-driven variables, 768 variables from Bio-Clinical BERT, and the 10 most relevant topic modeling variables in our final model.

Data were stratified into the training (80%) and test (20%) sets. Next, the final model was evaluated on the test set. We evaluated the predictive ability of models on the test set using the following criteria: recall, precision, accuracy, F1, and area under the receiver operating characteristic curve (AUC). Figure 1 provides a general overview of the study methods.

Figure 1.

Figure 1.

Open in a new tab

Overview of study methods.

In terms of missing data, our study utilized 3 types of variables: (1) OASIS variables, which are federally mandated and have a high completion rate of 99%+; (2) vital signs, of which unmeasured values were categorized as “not available”; and (3) NLP-derived variables, including rule-based, TF-IDF, Bio-Clinical BERT, and LDA, of which the absence of documentation resulted in categorization as “not available.”

Identifying the most highly correlated variables associated with risk for ED visits and hospitalizations

To identify both positively and negatively highly correlated variables with risk for ED visits and hospitalizations considering coefficients, we used an approach based on the least absolute shrinkage and selection operator (LASSO).42 One of the core strengths of the LASSO approach is the ability to identify the set of predictors associated with the outcome variable, subject to a constraint on the total size of the coefficients. The idea behind LASSO is to shrink the coefficients of less important predictors toward zero, eliminating them from the model and only including the strongest relationship to the outcome variable.42 We implemented LASSO using Python’s scikit-learn and presented the top 20 variables either positively or negatively associated with the risk of ED visits or hospitalizations using the optimal value of alpha.

RESULTS

Patient characteristics

In total, we identified 9362 patients diagnosed with HF who received 176 209 visits during 12 223 episodes of HHC. The characteristics of patients are listed in Table 1. A majority of the patients were female (61%) and on average 81.7 years old (standard deviation [SD] 11 years) at the start of care. The average length of stay in HHC was 48 days (SD 56 days). About 1 in 4 patients (2379/9362 = 25%) experienced hospitalization or ED visits within the 60-day period. The rule-based NLP algorithm identified documented symptoms in 41.5% (n = 3886) of patients. Frequently documented symptoms were dyspnea (17.5%), peripheral edema (13.7%), and fatigue (11.4%).

Table 1.

Characteristics of patients

Clinical and demographic profile (OASIS item) Total (n = 9362, 12 223 HHC episodes)
Patient n (%)a
Age at start of care (mean, years, SD) 81.7 (11)
Race
 Asian/Others/Unknown/Native Hawaiian or Pacific Islander 3498 (37)
 Non-Hispanic Black 1279 (14)
 Non-Hispanic White 3494 (37)
 Hispanic 1091 (12)
Gender
 Male 3644 (39)
Episode n (%)b
Length of stay in HHC (mean, days, SD) 47.6 (56)
Comorbidities (mark all that apply)
 AIDS 95 (1)
 Arthritis 1553 (13)
 Diabetes 4746 (39)
 Dementia 1338 (11)
 Hypertension 8851 (72)
 Peripheral vascular disease 444 (4)
 Pulmonary diseases 3151 (26)
 Renal diseases 726 (6)
 Skin ulcer 1631 (13)
Prior conditions within past 14 days
 Indwelling/suprapubic catheter 152 (1)
 Impaired decision-making 1285 (11)
 No inpatient facility discharge and no change in medical or treatment regimen 1105 (9)
 Urinary incontinence 3743 (31)
Risk factors
 Drug 71 (1)
 Obesity 2279 (19)
Shortness of breath
 Never 4039 (33)
 With exertion 7909 (65)
 At rest 183 (2)
Open in a new tab
a

The descriptions of demographic characteristics were analyzed at the patient level.

b

The descriptions of clinical characteristics were analyzed at the episode level.

Performance of risk prediction models

Risk prediction results using different sets of variables

With the addition of increasingly complex sets of variables, the risk prediction ability of the model improved. When we only used OASIS variables, the model had the lowest F1 score of 0.57 (95% confidence interval [CI]: 0.54, 0.6) (Figure 2). When we added rule-based NLP-derived variables, the F1 score of the model improved from 0.57 to 0.67 (10% improvement compared to baseline). Additional detailed metrics are reported in Table 2. Our findings show that our model can predict ED visits and hospitalizations using a 4-day time window with all 7 sets of variables with an F1 score of 0.69.

Figure 2.

Figure 2.

Open in a new tab

F1 score of risk prediction models when adding the different sets of variables. Set 1: OASIS only. Set 2: OASIS+vital signs. Set 3: OASIS+vital signs+visit characteristics. Set 4: OASIS+vital signs+visit characteristics+NLP variables. Set 5: OASIS+vital signs+visit characteristics+NLP variables+TF-IDF variables. Set 6: OASIS+vital signs+visit characteristics+NLP variables+TF-IDF variables+Bio-Clinical BERT variables. Set 7: OASIS+vital signs+visit characteristics+NLP variables+TF-IDF variables+Bio-Clinical BERT variables+topic modeling variables.

Table 2.

Four-day ED visit and hospitalization risk prediction performance

Sets of variables Recall Precision Accuracy F1 AUC
 OASIS data 0.69 [0.61, 0.77] 0.49 [0.44, 0.54] 0.79 [0.76, 0.82] 0.57 [0.54, 0.60] 0.82 [0.80, 0.84]
 Vital signs 0.66 [0.60. 0.78] 0.54 [0.47, 0.61] 0.82 [0.78, 0.84] 0.59 [0.57, 0.63] 0.84 [0.82, 0.86]
 Visit characteristics 0.66 [0.57, 0.79] 0.57 [0.48, 0.64] 0.83 [0.79, 0.87] 0.61 [0.58, 0.64] 0.85 [0.83, 0.87]
NLP technique-derived variables
 Rule-based NLP 0.73 [0.69, 0.77] 0.62 [0.58, 0.66] 0.86 [0.84, 0.86] 0.67 [0.64, 0.70] 0.88 [0.86, 0.90]
 TF-IDF 0.71 [0.67, 0.77] 0.63 [0.58, 0.68] 0.86 [0.84, 0.88] 0.67 [0.64, 0.70] 0.89 [0.87, 0.91]
 Bio-clinical BERT 0.76 [0.65, 0.81] 0.61 [0.56, 0.72] 0.85 [0.84, 0.88] 0.68 [0.65, 0.71] 0.90 [0.87, 0.91]
 LDA topic modeling (the number of topics: 10) 0.73 [0.66, 0.80] 0.65 [0.59, 0.73] 0.87 [0.85, 0.89] 0.69 [0.66, 0.72] 0.89 [0.87, 0.91]
Open in a new tab

Note: The best result on each metric is shown in bold.

Abbreviation: AUC: area under the receiver operating characteristic curve.

Risk prediction results using different time windows

Using the best predictive model with all 7 sets of variables, we tested its predictive ability in different time windows (1–15, 30, 45, and 60 days before the date of ED visit or hospitalization). Overall, we achieved relatively high and stable performance for predictions of ED visits or hospitalizations starting on day 4 (F1 score = 0.69; further details are provided in Supplementary Table S3), shown in Figure 3. Even though we observed the best performance using 7 days of data, there was only 0.8% improvement on the F1 score using 7 days compared to predictions using 4 days. Clinically, predictions within shorter time windows are more valuable; hence we decided to use the 4-day time window moving forward (Table 2). Additional details of prediction performance on the different days are in Supplementary Table S3.

Figure 3.

Figure 3.

Open in a new tab

F1 score of ED visit and hospitalization risk prediction for different time windows.

To further describe the performance of our model, we also applied the receiver-operating characteristic (ROC) curves and the precision-recall (PR) curves, shown in Figure 4. Our risk model had a relatively high AUC (0.89), and area under the PR curve (0.72), indicating good predictive performance.

Figure 4.

Figure 4.

Open in a new tab

Performance of prediction model to predict emergency department visits and hospitalizations within 4 days. (Left) Receiver-operating characteristic curves. (Right) precision-recall curves.

Highly correlated variables associated with risk for ED visits and hospitalizations

Figure 5 displays the top 20 variables either positively or negatively correlated with the outcome within the 4-day time window as identified by LASSO. Three variables related to HHC visit characteristics were identified. First, the “number of days since admission for the current visit” was the highest-ranked variable, signifying those patients with shorter HHC stays between their HHC admission and the current visit had a higher risk for negative outcomes. Similarly, the sixth-ranked variable, “the number of days between last visit and previous visit,” showed that patients with less time between visits faced increased risk. Moreover, the fifth-ranked variable, “visit purpose (for the current visit),” was linked to negative outcomes. A sub-analysis of categorical documented visit purposes revealed that patients with missed visits (eg, when not at home or not answering the door) had a higher likelihood of ED visits and hospitalizations.

Figure 5.

Figure 5.

Open in a new tab

Twenty variables associated with risk for ED visits and hospitalizations using LASSO. The x axis represents the log of the L1 penalty parameter (alpha), and the y axis represents the coefficient values of the predictors in the model. The L1 penalty parameter shows the strength of the regularization, and as the value of alpha increases, the coefficients shrink toward zero. Each line in the plot represents a different predictor in the model, and the slope of the line represents the change in the magnitude of the coefficient as alpha increases. Predictors with nonzero coefficients at high values of alpha are considered more important, while predictors with zero coefficients are less important. We generated several variables of TF-IDF, describing lexical features of the text, including the “ratio of nonalphanumeric symbols to text length” and “ratio of numeric digits to text length.”

Next, 14 NLP-extracted variables were identified. Three of these variables were indicators of the “total number of HF symptoms” at previous visit, 12, and 25 visits previously (second, seventh, and 13th ranked variables, respectively). Interestingly, this directionality of association for this variable changed over time. Specifically, having more HF symptoms at a previous HHC visit indicated higher risk, whereas having more HF symptoms at visits that happened a while ago (ie, 12 and 25 visits previously) indicated lower risk. Other important NLP-extracted variables included 2 Bio-Clinical BERT-derived variables (not explainable) and 4 tokens extracted from the TF-IDF vector, specifically “ER [emergency room]” and “has pain” were associated with higher risk, whereas “no further” and “generic” were associated with lower risk. The third-ranked feature was the rule-based NLP-derived variable, “community resources at previous visit,” showing that the fewer total number of risk factors related to “community resources” documented in clinical notes during the previous visit is also associated with a higher risk. Two lexical variables were associated with risk: “ratio of nonalphanumeric symbols to text length” and “ratio of numeric digits to text length” (fourth, and 17th ranked variables, respectively). Two variables describing topics identified by topic modeling were identified as associated with higher risk, including the presence of “Referral related language at current visit” and “Comorbidity management at previous visit” (18th and 19th ranked variables, respectively).

Finally, 3 OASIS variables were identified as associated with increased risk, namely “[lower level of] Prior functioning,” “Skin ulcer,” and “Diabetes.” Of note, no vital signs were selected among the top variables associated with risk.

DISCUSSION

This study generated a time series risk model to predict ED visit and hospitalization risk in patients with HF. The novelty of a time series risk model to predict ED visit and hospitalization in patients with HF is in its ability to analyze data over time and account for the dynamic nature of the disease, assisting HHC providers to identify these changes and be alerted to intervene early, potentially preventing an ED visit or hospitalization. This is the first study in HHC to use all available data over the episode of care to generate risk models. Specifically, we extended the rigor of previous research in HHC that primarily relied on standardized assessments, such as an OASIS, for risk prediction tasks.43,44 We added information extracted from structured data (including vital signs and visit characteristics) and NLP-derived variables to our risk models. We found that gradually adding different variable sets improves risk prediction performance. In line with previous research,16,45 we found that the F1 score improved the most when we included rule-based NLP-derived variables (an improvement of 10% compared to using only OASIS-based risk prediction). This result demonstrates that more information is captured in the clinical notes. This finding indicates that HHC risk prediction models can be improved by including a wide array of risk factors, including data extracted from administrative sources and clinical notes.

To minimize the risk of negative outcomes, it is essential for HHC providers to quickly identify deteriorating patients to provide early interventions before they need to be hospitalized or visit the ED.14 In the hospital setting, early risk identification models accurately identify patients at risk for negative outcomes as early as 24 hours before the event.46 In HHC, our study achieved relatively high and stable risk prediction performance 4 days before the outcome. This gap in risk prediction windows between HHC and hospitals might be partially explained by the frequency of data collection. In hospitals, patient data is collected very frequently—sometimes every second (eg, continuous patient monitoring).46,47 However, in the HHC setting, observations and new data points are generated much less frequently, as HHC visits typically occur every 2–4 days on an average.41 Additionally, the mean length of stay for patients was shorter by one-third (3.2 vs 4.9 days) in HHC compared to hospital settings.48 Hence, less frequently collected data in HHC allowed us to build risk prediction with a longer time window of 4 days or longer compared to hospital settings.

Previous studies have used various time windows to predict ED visits or hospitalizations in HHC, ranging from 30 days44,49 to 60 days.50,51 In this study, we discovered that using a 4-day time window for risk prediction produces adequate risk models. On one hand, identifying risk at this time window could help HHC providers intervene and prevent negative outcomes. On the other hand, a shorter risk time window may help identify patients who need immediate attention due to rapid deterioration. For example, HHC providers could use the identified features to guide their decision-making around when to visit or call the physician, and to conduct more thorough assessments within a 4-day time window after the patient’s HHC admission. Additionally, healthcare providers could use the identified features to guide patient education around HF self-management, including symptom recognition, dietary guidelines, medication management, and weight monitoring. This education could be based on randomized controlled trials of home nursing visits for HF to ensure that patients receive the best possible care.52 More research is needed to determine the best risk time window for HHC settings.

Another significant and innovative contribution of this study is identifying risk factors associated with time series risk for ED visits and hospitalizations in the HHC setting. Applying the LASSO variable selection technique, we found that visit characteristics collected over the previous HHC visits correlate highly with the patient’s risk. Specifically, we found that shorter HHC episodes and shorter times between the current and previous HHC visits were highly correlated with the risk of ED visits and hospitalizations. This is not surprising; more frequent clinician visits often correlate with patients’ clinical complexity or deterioration in patient health status.53 These findings are similar to those from the hospital setting, where having a shorter interval between assessments by clinicians was found to be early deterioration signals.54,55 Previous research also shows that patients with episodes less than 21 days were more likely to be readmitted.41 This finding highlights the importance of providing timely interventions, including comprehensive assessments, education, and management of HF symptoms within a specific time frame during the HHC episode to prevent future ED visits or hospitalizations. We also found that missed HHC visits are associated with higher risk, consistent with previous studies that show that missing or refusing HHC services increases patient risk.56 Further risk prediction modeling in HHC should strongly consider using care patterns and visit characteristics.

We also found that multiple rule-based NLP-derived variables correlate highly with patient risk. The first set of NLP variables is a total number of specific HF-related symptoms extracted from clinical notes at previous HHC visits. Our previous work shows that this is an important factor in HHC episode-level risk prediction,15,16,18 and this study confirms its importance in time series risk modeling. Interestingly, our current results extend the previous research by identifying that the total number of HF symptoms at the previous visit indicated increased risk. In contrast, the same variable indicated lower risk when observed during visits that occurred some time ago. This might further imply that recent symptom documentation indicates risk, whereas earlier documentation might pertain to symptoms that have since been addressed and managed, thus correlating with a lower risk.

In addition, we found that topic models indicating comorbidity management or referral language are associated with higher risk. These findings further advance our HHC episode-level insights showing that the presence of health service use is a significant risk indicator.57 We also found several Bio-Clinical BERT variables were associated with risk; these variables are not easily explainable to HHC clinicians. Further, several words were identified based on their TF-IDF values, including high-risk words like “ER” (which often indicates previous ED visits) and “has pain” and low-risk words like “no further” (which often indicates that no further HHC is needed) and “generic” as variables associated with risk for ED visit and hospitalization. Creating a clinician-interpretable risk prediction model is essential for clinical adoption and implementation of models because it builds trust in decision-makers, enables error identification and correction in the model, and facilitates integration into clinical workflows.58 Further research is needed to understand how to best present these risk factors to HHC providers.

In hospitals, trends in vital signs often offer strong signals for risk prediction, and some hospital-based risk models mainly rely on these routinely collected measurements.46 Surprisingly, vital signs were not selected as the top variables highly correlated with HHC patient risk in this study. This might indicate that vital signs collected every few days in HHC offer less signal for risk prediction than in hospitals, where vital signs are collected frequently (eg, hourly). The patient with HF might have abnormal vital signs at baseline or HHC admission, so it might be more important if the vital signs changed or worsened and how much they changed than whether they were normal or abnormal. Additionally, sudden or gradual changes in vital signs are often one of the last signals to show up before deterioration in patients with HF59 and therefore, the visit-level time window might be too wide to pick them up. Further research is needed to understand why vital signs in HHC offer little risk prediction value for patients with HF and to further utilize patterns of vital sign changes over time with a shorter time window.

This prediction model suggests that HHC clinicians can identify patients with HF at risk for visiting the ED or being hospitalized 4 days before the event, allowing clinicians to deliver earlier, more targeted interventions. For example, HHC nurses could use the identified risk factors to guide their decision-making about when to call the patient’s physician or conduct more thorough clinical assessments. Further risk prediction modeling in HHC should consider using care patterns, visit characteristics, and clinical notes, as these were among the most important features associated with a high risk of ED visits or hospitalizations. Early interventions can be triggered to prevent these negative outcomes through clinical decision support modules integrated into EHR systems. Further studies are needed to explore possible clinical decision support applications in HHC that can improve patient outcomes, optimize resource allocation, and enhance the quality of care delivered for patients with HF.

Study limitations

This study has several important limitations. The study sample was drawn from a single, albeit large, HHC organization in New York City, which may limit the generalizability of the findings to other locations. The study focused on patients with HF, and the findings may not apply to other patient populations. When running risk prediction models with different combinations of data sets, we did not consider the order in which specific data sets were added. Specifically, the improvement of 2 percentage points in the F1 from sets of variables 4 (0.67) to 7 (0.69) is limited to justify all the extra work of obtaining and using sets of variables 5, 6, and 7. We recognize that our study’s data, collected from 2015 to 2017, may be considered outdated. Although this is a limitation, the data was collected rigorously, and the methods remain pertinent and valuable for the HHC setting. To address this limitation, future research should use more recent data. Further, more advanced machine learning models might achieve better risk prediction results.

Clinical implications and further research

The risk prediction model developed in this study can help provide more targeted treatment for patients with HF in HHC, helping to better manage symptoms and risk factors.60 Further research is needed to inform the clinical implementation of such models in HHC, tailoring these models to the needs of HHC providers and patients with HF. Some central questions remain about presenting the risk score to HHC providers, explaining certain risk factors (eg, Bio-Clinical BERT variables), applying different advanced machine learning models, and developing sets of interventions to prevent ED visits and hospitalizations.

CONCLUSIONS

In conclusion, this study demonstrated the feasibility of using routinely collected HHC data to develop a time series ED visit and hospitalization risk prediction model for patients with HF. The risk model was built on a combination of structured and unstructured datasets and visit characteristics and rule-based NLP-derived variables were highly correlated with patients’ risk. The ability to predict negative outcomes would allow for more targeted treatment and better management of symptoms and risk factors in this patient population. Further research is needed to understand how to apply this risk model in HHC practice.

Supplementary Material

ocad129_Supplementary_Data
Click here for additional data file. (81.4KB, docx)

Contributor Information

Sena Chae, College of Nursing, The University of Iowa, Iowa City, Iowa, USA.

Anahita Davoudi, Center for Home Care Policy & Research, VNS Health, New York, New York, USA.

Jiyoun Song, Columbia University School of Nursing, New York City, New York, USA.

Lauren Evans, Center for Home Care Policy & Research, VNS Health, New York, New York, USA.

Mollie Hobensack, Columbia University School of Nursing, New York City, New York, USA.

Kathryn H Bowles, Center for Home Care Policy & Research, VNS Health, New York, New York, USA; Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, USA.

Margaret V McDonald, Center for Home Care Policy & Research, VNS Health, New York, New York, USA.

Yolanda Barrón, Center for Home Care Policy & Research, VNS Health, New York, New York, USA.

Sarah Collins Rossetti, Columbia University School of Nursing, New York City, New York, USA; Department of Biomedical Informatics, Columbia University, New York City, New York, USA.

Kenrick Cato, Department of Family and Community Health, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, USA.

Sridevi Sridharan, Center for Home Care Policy & Research, VNS Health, New York, New York, USA.

Maxim Topaz, Center for Home Care Policy & Research, VNS Health, New York, New York, USA; Columbia University School of Nursing, New York City, New York, USA; Data Science Institute, Columbia University, New York City, New York, USA.

FUNDING

This study was funded by Agency for Healthcare Research and Quality (AHRQ) (R01 HS027742), “Building risk models for preventable hospitalizations and emergency department visits in homecare (Homecare-CONCERN).” The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. Ms Hobensack is supported by the National Institute for Nursing Research training grant Reducing Health Disparities through Informatics (RHeaDI) (T32NR007969) as a predoctoral trainee and the Jonas Scholarship.

AUTHOR CONTRIBUTIONS

SC conducted variable identification, data extraction, statistical analysis, and drafting of the work. AD conducted model development, statistical analysis, and drafting of the work. SC, JS, and MH conducted rule-based NLP variable selection. AD, JS, LE, MH, KHB, MVM, YB, KC, SC, SS, and MT helped to design experiments and revise the drafted manuscript. All authors approved the submitted version. SC and AD are equally contributed, first authors.

ETHICS APPROVAL

This study was approved by the VNS Health Institutional Review Board (IRB No. I20-003).

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The data underlying this article cannot be shared publicly due to the privacy protection requirements of patient healthcare data.

REFERENCES

  • 1. Wang Y, Spatz ES, Tariq M, Angraal S, Krumholz HM.. Home health agency performance in the United States: 2011–15. J Am Geriatr Soc 2017; 65 (12): 2572–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Sterling MR, Kern LM, Safford MM, et al. Home health care use and post-discharge outcomes after heart failure hospitalizations. JACC Heart Fail 2020; 8 (12): 1038–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Kavalieratos D, Gelfman LP, Tycon LE, et al. Palliative care in heart failure. J Am Coll Cardiol 2017; 70 (15): 1919–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Bekelman DB, Havranek EP, Becker DM, et al. Symptoms, depression, and quality of life in patients with heart failure. J Card Fail 2007; 13 (8): 643–8. [DOI] [PubMed] [Google Scholar]
  • 5. Jackson SL, Tong X, King RJ, Loustalot F, Hong Y, Ritchey MD.. National burden of heart failure events in the United States, 2006 to 2014. Circ Heart Fail 2018; 11 (12): e004873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Lee DS, Stukel TA, Austin PC, et al. Improved outcomes with early collaborative care of ambulatory heart failure patients discharged from the emergency department. Circulation 2010; 122 (18): 1806–14. [DOI] [PubMed] [Google Scholar]
  • 7. Bennett SJ, Huster GA, Baker SL, et al. Characterization of the precipitants of hospitalization for heart failure decompensation. Am J Crit Care 1998; 7 (3): 168–74. [PubMed] [Google Scholar]
  • 8. Friedman MM. Older adults’ symptoms and their duration before hospitalization for heart failure. Heart Lung 1997; 26 (3): 169–76. [DOI] [PubMed] [Google Scholar]
  • 9. Farré N, Vela E, Clèries M, et al. Real world heart failure epidemiology and outcome: a population-based analysis of 88,195 patients. PLoS One 2017; 12 (2): e0172745. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Ziaeian B, Fonarow GC.. The prevention of hospital readmissions in heart failure. Prog Cardiovasc Dis 2016; 58 (4): 379–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Dunbar-Yaffe R, Stitt A, Lee JJ, Mohamed S, Lee DS.. Assessing risk and preventing 30-day readmissions in decompensated heart failure: opportunity to intervene? Curr Heart Fail Rep 2015; 12 (5): 309–17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Jonkman NH, Westland H, Groenwold RHH, et al. Do self-management interventions work in patients with heart failure? Circulation 2016; 133 (12): 1189–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Hobensack M, Song J, Scharp D, Bowles KH, Topaz M.. Machine learning applied to electronic health record data in home healthcare: a scoping review. Int J Med Inform 2023; 170: 104978. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Hobensack M, Ojo M, Barrón Y, et al. Documentation of hospitalization risk factors in electronic health records (EHRs): a qualitative study with home healthcare clinicians. J Am Med Inform Assoc 2022; 29 (5): 805–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Song J, Ojo M, Bowles KH, et al. Detecting language associated with home healthcare patient’s risk for hospitalization and emergency department visit. Nurs Res 2022; 71 (4): 285–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Song J, Hobensack M, Bowles KH, et al. Clinical notes: an untapped opportunity for improving risk prediction for hospitalization and emergency department visit during home health care. J Biomed Inform 2022; 128: 104039. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Chae S, Song J, Ojo M, Topaz M.. Identifying heart failure symptoms and poor self-management in home healthcare: a natural language processing study. Stud Health Technol Inform 2021; 284: 15–9. [DOI] [PubMed] [Google Scholar]
  • 18. Chae S, Song J, Ojo M, et al. Factors associated with poor self-management documented in home health care narrative notes for patients with heart failure. Heart Lung 2022; 55: 148–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Gasparrini A. The case time series design. Epidemiology 2021; 32 (6): 829–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. World Health Organization. ICD-10: International Statistical Classification of Diseases and Related Health Problems: Tenth Revision. 2nd ed. Geneva: World Health Organization; 2010. [Google Scholar]
  • 21. Medicare Payment Advisory Commission. Report to the congress: Medicare Payment Policy. Washington, DC: Medicare Payment Advisory Commission; 2019.
  • 22. Blecker S, Ladapo JA, Doran KM, Goldfeld KS, Katz S.. Emergency department visits for heart failure and subsequent hospitalization or observation unit admission. Am Heart J 2014; 168 (6): 901–8. e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Abraham WT, Trupp RJ, Mehra MR, et al. Prospective evaluation of cardiac decompensation in patients with heart failure by impedance cardiography test: the predict multicentertrial. Circulation 2004; 110(Suppl III): 597. [Google Scholar]
  • 24. Greenland P, Peterson E.. The new 2017 ACC/AHA guidelines “up the pressure” on diagnosis and treatment of hypertension. JAMA 2017; 318 (21): 2083–4. [DOI] [PubMed] [Google Scholar]
  • 25. Tavabi N, Singh M, Pruneski JA, Kiapour A.. Systematic evaluation of common natural language processing techniques to codify clinical notes. Published Online First: 10 October 2022. https://www.medrxiv.org/content/10.1101/2022.10.10.22280852v1. Accessed July 03, 2023. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Topaz M, Radhakrishnan K, Blackley S, Lei V, Lai K, Zhou L.. Studying associations between heart failure self-management and rehospitalizations using natural language processing. West J Nurs Res 2017; 39 (1): 147–65. [DOI] [PubMed] [Google Scholar]
  • 27. Topaz M, Murga L, Bar-Bachar O, McDonald M, Bowles K.. NimbleMiner: an open-source nursing-sensitive natural language processing system based on word embedding. Comput Inform Nurs 2019; 37 (11): 583–90. [DOI] [PubMed] [Google Scholar]
  • 28. Ramos J. Using TF-IDF to determine word relevance in document queries. In: Proceedings of the First Instructional Conference on Machine Learning; Citeseer; 2003: 242 (1): 29–48.
  • 29. Lubis AR, Nasution MK, Sitompul OS, Zamzami EM.. The effect of the TF-IDF algorithm in times series in forecasting word on social media. Indones J Electr Eng Comput Sci 2021; 22 (2): 976. [Google Scholar]
  • 30. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. Published Online First: 24 May 2019. https://arxiv.org/abs/1810.04805. Accessed July 03, 2023.
  • 31. Google AI Blog. Open sourcing BERT: state-of-the-art pre-training for natural language processing. https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html. Accessed January 9, 2023.
  • 32. Alsentzer E, Murphy JR, Boag W, et al. Publicly available clinical BERT embeddings. Published Online First: 20 June 2019. https://arxiv.org/abs/1904.03323. Accessed July 03, 2023.
  • 33. Kshatriya BSA, Nunez NA, Resendez MG, et al. Neural language models with distant supervision to identify major depressive disorder from clinical notes. Published Online First: 19 April 2021. https://arxiv.org/abs/2104.09644. Accessed July 03, 2023.
  • 34. Blei DM, Ng AY, Jordan MI.. Latent Dirichlet allocation. J Mach Learn Res 2003; 3 (Jan): 993–1022. [Google Scholar]
  • 35. Kumar L, Greiner R.. Gene expression based survival prediction for cancer patients—a topic modeling approach. PLoS One 2019; 14 (11): e0224446. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Breslow NE. Analysis of survival data under the proportional hazards model. Int Stat Review 1975; 43 (1): 45–57. [Google Scholar]
  • 37. Erickson N, Mueller J, Shirkov A, et al. AutoGluon-tabular: robust and accurate AutoML for structured data. Published Online First: 13 March 2020. https://arxiv.org/abs/2003.06505. Accessed July 03, 2023.
  • 38. Hutter F, Lücke J, Schmidt-Thieme L.. Beyond manual tuning of hyperparameters. Künstl Intell 2015; 29 (4): 329–37. [Google Scholar]
  • 39. Michaud A, Parker SIA, Ganshorn H, Ezekowitz JA, McRae AD.. Prediction of early adverse events in emergency department patients with acute heart failure: a systematic review. Can J Cardiol 2018; 34 (2): 168–79. [DOI] [PubMed] [Google Scholar]
  • 40. Murtaugh CM, Deb P, Zhu C, et al. Reducing readmissions among heart failure patients discharged to home health care: effectiveness of early and intensive nursing services and early physician follow-up. Health Serv Res 2017; 52 (4): 1445–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. O’Connor M, Hanlon A, Naylor MD, Bowles KH.. The impact of home health length of stay and number of skilled nursing visits on hospitalization among medicare-reimbursed skilled home health beneficiaries. Res Nurs Health 2015; 38 (4): 257–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Tibshirani R. Regression shrinkage and selection via the LASSO. J R Stat Soc B Stat Methodol 1996; 58 (1): 267–88. [Google Scholar]
  • 43. Lo Y, Lynch SF, Urbanowicz RJ, et al. Using machine learning on home health care assessments to predict fall risk. Stud Health Technol Inform 2019; 264: 684–688. [DOI] [PubMed] [Google Scholar]
  • 44. Jones CD, Falvey J, Hess E, et al. Predicting hospital readmissions from home healthcare in medicare beneficiaries. J Am Geriatr Soc 2019; 67 (12): 2505–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Seinen TM, Fridgeirsson EA, Ioannou S, et al. Use of unstructured text in prognostic clinical prediction models: a systematic review. J Am Med Inform Assoc 2022; 29 (7): 1292–302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Gerry S, Bonnici T, Birks J, et al. Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ 2020; 369: m1501. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Ghosh E, Eshelman L, Yang L, Carlson E, Lord B.. Description of vital signs data measurement frequency in a medical/surgical unit at a community hospital in United States. Data Brief 2018; 16: 612–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48. Leff B, Burton L, Mader SL, et al. Hospital at home: feasibility and outcomes of a program to provide hospital-level care at home for acutely ill older patients. Ann Intern Med 2005; 143 (11): 798–808. [DOI] [PubMed] [Google Scholar]
  • 49. O’Connor M, Hanlon A, Bowles KH.. Impact of frontloading of skilled nursing visits on the incidence of 30-day hospital readmission. Geriatr Nurs 2014; 35 (2 Suppl): S37–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Shang J, Russell D, Dowding D, et al. A predictive risk model for infection-related hospitalization among home healthcare patients. J Healthc Qual 2020; 42 (3): 136–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51. Rosati RJ, Huang L.. Development and testing of an analytic model to identify home healthcare patients at risk for a hospitalization within the first 60 days of care. Home Health Care Serv Q 2007; 26 (4): 21–36. [DOI] [PubMed] [Google Scholar]
  • 52. Feltner C, Jones CD, Cené CW, et al. Transitional care interventions to prevent readmissions for persons with heart failure: a systematic review and meta-analysis. Ann Intern Med 2014; 160 (11): 774–84. [DOI] [PubMed] [Google Scholar]
  • 53. Rossetti SC, Knaplund C, Albers D, et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework. J Am Med Inform Assoc 2021; 28 (6): 1242–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54. Schnock KO, Kang MJ, Rossetti SC, et al. Identifying nursing documentation patterns associated with patient deterioration and recovery from deterioration in critical and acute care settings. Int J Med Inform 2021; 153: 104525. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55. Keim-Malpass J, Clark MT, Lake DE, Moorman JR.. Towards development of alert thresholds for clinical deterioration using continuous predictive analytics monitoring. J Clin Monit Comput 2020; 34 (4): 797–804. [DOI] [PubMed] [Google Scholar]
  • 56. Topaz M, Kang Y, Holland DE, Ohta B, Rickard K, Bowles KH.. Higher 30-day and 60-day readmissions among patients who refuse post acute care services. Am J Manag Care 2015; 21 (6): 424–33. [PMC free article] [PubMed] [Google Scholar]
  • 57. Topaz M, Woo K, Ryvicker M, Zolnoori M, Cato K.. Home healthcare clinical notes predict patient hospitalization and emergency department visits. Nurs Res 2020; 69 (6): 448–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58. Markus AF, Kors JA, Rijnbeek PR.. The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform 2021; 113: 103655. [DOI] [PubMed] [Google Scholar]
  • 59. Prgomet M, Cardona-Morrell M, Nicholson M, et al. Vital signs monitoring on general wards: clinical staff perceptions of current practices and the planned introduction of continuous monitoring technology. Int J Qual Health Care 2016; 28 (4): 515–21. [DOI] [PubMed] [Google Scholar]
  • 60. Albert NM, Barnason S, Deswal A, et al. Transitions of care in heart failure. Circ Heart Fail 2015; 8 (2): 384–409. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ocad129_Supplementary_Data
Click here for additional data file. (81.4KB, docx)

Data Availability Statement

The data underlying this article cannot be shared publicly due to the privacy protection requirements of patient healthcare data.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES