Detecting Smartwatch-Based Behavior Change in Response to a Multi-Domain Brain Health Intervention

To characterize a person's overall behavior routine, we extract digital behavior markers from sensor data that are automatically labeled with corresponding activity categories. Performing human activity recognition from wearable sensors has become a popular topic for researchers to investigate [6, 7]. Because we can collect uninterrupted data in all locations each person visits, wearable sensors are a natural choice for this investigation. Earlier approaches to activity recognition consider many diverse learning models, including decision trees, nearest neighbors, clusters, and classifier ensembles [8–10]. With the rising popularity of deep learning, these methods have also been explored for their ability to learn deep features that are useful in expressing sensed activities [11–14]. The field has matured to the point that numerous public datasets are now available for comparative analysis of recognition methods [11].

Limitations with many of these existing methods are that they focus on basic, repetitive movement types and are typically evaluated in laboratory settings. Recognizing postures and ambulatory motions such as sitting, standing, climbing, lying down, and running are useful for monitoring gait characteristics and activity levels. However, we are interested in determining change to a person's entire behavioral routine. This is frequently characterized by more complex patterns, such as basic and instrumental activities of daily living (iADLs) [15, 16]. A few researchers consider activity categories that combine ambulation with context (e.g., standing indoors versus outdoors) [8, 17, 18]. They accomplish this by adding location information. This location information is provided by ambient motion sensors inside a specific physical environment [17] or a grid location within a specific physical region [8]. A challenge with analyzing specific locations is that the insights do not generalize well to new individuals. Lin and Hsu [19] introduced generalizable location features that we employ in our method, including heading change rate, distance covered, and velocity change rate. We further integrate ideas from Boukhechba et al. [20], who cluster location readings to detect person-specific frequented spots. Although we extend these previous methods to perform real-time activity labeling of smartwatch sensor data in naturalistic settings, the success these researchers have documented in recognizing a wide assortment of activities suggests the general applicability of activity learning and labeling for smartwatch data.

Sensor data offer substantial insights on a person's behavior as well as their health. Now that wearable sensors are decreasing in cost and are deployable in real-world settings, researchers have discovered that they can link behavior interventions with specific, sensor-observed target behavior outcomes. As wearable sensor systems also become more ubiquitous, researchers have increasingly used these platforms to impact and measure change in human behavior. Typically, these prior works target a specific behavior. Popular target behaviors include activity levels [21, 22] and sleep [23]. In these cases, change can be quantified by a single variable such as step counts, sleep duration, or a clinical score [24]. Jain et al. [25] designed a method to analyze the univariate change in the state of a person or place, such as the amount of activity in a region of the home. Wang et al. [26] calculated the variability of a single variable, then joined the variability of multiple features into a single vector to predict personality traits. In a method that utilizes mobile platforms for intervention delivery as well as evaluation, Costa et al. [27] delivered haptic feedback to participants intended to slow their heartbeats while completing math tests. They observed impacts based on self-reported anxiety, sensor-measured heart rate variability, and test performance. In a study conducted by Stanley et al. [28], individuals with dementia received coaching in behavior techniques designed to reduce anxiety. Here, outcomes were measured through clinician-rated and collateral report clinical scores targeted at assessing anxiety. Additional studies monitor single wearable markers to assess intervention impact for target conditions such as substance use disorder [29] and diabetes management [30].

In other work, researchers targeted specific sensor-observed behavior markers as a mechanism for assessing the relationship between lifestyle and health. Specifically, Dhana et al. [31] quantify healthy behavior as a combination of non-smoking, physical activity, alcohol consumption, nutrition, and cognitive activities. Individuals who scored higher on this behavior metric had lower risk of Alzheimer dementia. Other researchers have also found that sensor-based behavior patterns are predictive of cognitive health [32, 33]. Li et al. [33] found that physical activity level was predictive of Alzheimer's disease, whereas Aramendi et al. [32] were able to predict cognitive health and mobility from activity-labeled sensor data.

These studies provide evidence that wearable sensors afford the ability to monitor intervention impact and to assess a person's cognitive health. Within this area of investigation, our proposed approach is unique because we investigate a computational method to analyze intervention impact on the pattern of a person's activity-based routine. This holistic approach to sensor analysis of behavior is motivated by the design of a holistic, customized, sustainable healthy behavior intervention.

A critical component of this work is automatically detecting and quantifying change in behavior data observed longitudinally. Multiple options are available for computing change between two windows of multivariate time series data [34]. Supervised methods can classify a pair of time windows as “no change” or “change.” Such methods can also identify changes of a specific nature (e.g., a change from sedentary to active behavior), but these algorithms must be supplied with a sufficient amount of labeled training data [35]. Unsupervised approaches have been introduced as well. Some methods rely on pre-designed parametric models [36, 37], whereas others introduce more flexible non-parametric variations by directly estimating the ratio of probability densities for the two samples. In these cases, a larger ratio implies a greater likelihood that a change occurred between the two samples. Methods that fit in this category are KLIEP, which employs Kullback-Leibler (KL) divergence as a ratio estimator [38]; uLSIF, which employs Pearson divergence [38]; and Relative uLSIF (RuLSIF) [38]. In this work, we utilize a Permutation-Based Change Detection (PCD) method. This method is not only suited to comparing larger windows of sensor data but also incorporates a novel permutation mechanism to determine the significance of change over time.

An accumulating body of research suggests that adopting preventative health behaviors may promote healthy brain aging and slow cognitive decline [39]. Given that the etiology of dementia is heterogeneous and influenced by multiple risk factors, several recent large-scale prevention trials have focused on evaluation of multi-domain interventions [40, 41]. However, these interventions are currently limited by expensive treatment regimens and highly prescriptive goals that are difficult to sustain post-intervention. Furthermore, research has found that intervention success is promoted by healthy behavior education as well as designing interventions that are transferable out of the lab and into daily lives [42, 43]. In contrast with prior methods, the pilot group BHI involved creating individualized behavior change goals across multiple risk factor domains that could be integrated into the everyday lifestyles of participants and more easily sustained over time.

Our BHI, called B-Fit, is a 7-week intervention focused on providing brain health psychoeducation combined with individual behavior goal setting, group problem solving, and social support. The primary pilot clinical intervention study included three conditions: B-Fit, education-only, and waitlist [5]. Of the 68 participants allocated to the B-Fit intervention in the primary study, 28 of the participants wore a smartwatch during the intervention. These individuals were members of three separate B-Fit groups. The B-Fit intervention was delivered by two clinicians with groups of 8 to 13 participants. One learning topic was introduced to the group each week. During didactics, information about each topic was discussed with an emphasis on empirical research describing mechanisms by which lifestyles changes can improve brain health (e.g., reduce blood pressure and oxidative stress, increase cerebral blood flow) and contribute to a healthy aging phenotype. This approach is based on literature indicating the importance of linking healthy lifestyle behaviors with brain health and prevention research [54, 55]. The goal of this work is to introduce and validate a mechanism for assessing broad-spectrum lifestyle changes associated with this individualized, multi-domain intervention.

Table 1 lists the primary health topics covered during the weekly sessions. Each week, following the educational material presentation, participants developed a plan to integrate a personalized, realistic, sustainable, and positive lifestyle change into their everyday life (e.g., take a 10-minute walk daily after lunch). Health goals were built across multiple lifestyle domains consistent with the education topics (e.g., diet, socialization, sleep). Because a new goal was introduced each week, participants were assisted with developing goals for each topic that could be completed in about 5 to 10 minutes. At the beginning of each session, for each defined goal, participants indicate whether they made no change (0), partly met (1), completely met (2), or exceeded (3) their goal(s). Twenty-six of the 28 participants who wore a smartwatch regularly reported these values. The average self-reported adherence over the 8-week intervention was 1.7 (range [0.5, 2.3], standard deviation 0.44).

In addition to evaluating self-reported outcomes, determining the impact of interventions such as B-Fit on the pattern of behaviors can be implemented by passively and continuously monitoring behavior across time. Automated approaches to such ecologically valid monitoring and assessment pave the way for designing more effective treatment and quantitatively measuring the outcome of pharmaceutical and behavioral interventions. This ability will be particularly valuable in rural locations where access to the clinic or lab is limited [56]. Ecologically valid monitoring implies that a person's routine behavior is observed in naturalistic settings rather than asking individuals to perform scripted activities in controlled settings. This is particularly important for the B-Fit intervention because we want to assess the impact of the intervention on a person's entire set of activities, not on a few selected markers. Currently, holistically assessing change in overall behavior patterns in such a manner is difficult outside of manually observing a person's daily routine. We determine whether behavior change exists using a computational behavior change detection method. We further evaluate the pattern of behavior change over the course of the intervention.

To provide objective behavior routine and change measures, we gave 28 of the B-Fit participants two Apple Watches to wear continuously—one during the day and one at night (one watch was charged while the other was worn). We collected timestamped readings at 10 Hz for the following sensors: 3D acceleration, 3D rotation and rotation rate, course, speed, and 3D location. Participants received two motivational quotes through the smartwatch at random times during the day to support their lifestyle changes. Data were continuously collected at baseline (before intervention) and each week of the intervention. All data were collected before quarantine restrictions due to COVID-19.

All study participants are community-dwelling adults. We recruited participants from eastern Washington and northern Idaho. Exclusion criteria included a clinical diagnosis of dementia; inability to provide own consent; unstable or severely disabling disease (e.g., organ failure); and inability to complete assessment and intervention protocols due to communication, vision, hearing, or other medical difficulties. Prior to participation, participants were screened by phone with a medical/health interview and the Telephone Interview of Cognitive Status (TICS) [57]. Participants were excluded if they fell in the Impaired range on the TICS (score <26) or met study exclusion criteria. The study was approved by the Washington State University Institutional Review Board.

We analyze data for 28 study participants who received the active brain health education and goal-setting intervention (B-Fit) described in Section 2.1 and continuously wore smartwatches. This group is labeled “BHI.” To provide a comparison (non-intervention) group for behavior change detection, we analyze continuous smartwatch sensor data collected for 17 additional community-dwelling older adults who did not differ in age, t(43) = 1.08, p = .28, education, t(43) = .17, p = .87, or sex, X² (1, N = 45) = .33, p = .57, from the BHI group. This group is labeled “NI.” These participants were older adults who transformed their homes into smart homes and participated in a non-intervention, monitoring study designed to help validate smart home health algorithms. Table 2 summarizes the demographics for these participants.

Participants were given a white watch to wear during the day and a black watch to wear at night, affording time to charge the devices. The smartwatches indicate when they are on and off the charging station, allowing us to determine when data should be analyzed from one device or the other. However, participants were not consistent in wearing the designed watches during the specified times. To ensure we analyze the appropriate data, we examine the acceleration and rotation-based movement throughout each day. We merge day and night data collections into one by identifying “swapping” periods, or time points at which the subject swapped one watch for another. This was defined as a change point in the time series when one watch was at rest (total movement below a threshold amount) and the other was active. We create one merged set of sensor readings for each day by incorporating data at each time unit from the active watch.

Additionally, gaps occurred during data collection. Data for participants were 94.24% complete. Gaps in the data collection occurred during times when subjects forgot to charge and wear the watch. In these cases, we impute missing feature values based on the median value for the particular sensor given the day of the week, time of day, and neighboring context (readings before and after the gap).

We automatically label each set of sensor readings with a corresponding activity name. Activity labeling offers several benefits for our analysis. First, activities provide a vocabulary for expressing human behavior. From activity-labeled sensor data, we can see how a person spends their day and observe changes in these activity routines over time. Second, given activity-labeled data, we can target specific activity categories, such as work, exercise, and sleep, to ascertain whether the amount of time that is spent on these specific activities is changing, if the location where they occur changes, and if they occur at regular times.

We label sensor data using an activity recognition algorithm. Activity recognition maps a sequence of sensor readings onto an activity label. A time series data stream is an infinite sequence of elements \(S=\{x_{1},\ldots ,x_{i},\ldots\) , where x_i is a d-dimensional data vector arriving at timestamp i. Given a set of pre-defined activity categories, A, an activity recognition algorithm maps a sequence of time series data points \(\{x_{m},\ldots ,x_{k},\ldots , x_{n}\) , observed as part of time series S, onto an activity label a_j ∈ A. We map a fixed-length sequence of elements onto the activity label using the process illustrated in Figure 2.

To ensure that we label and monitor complex activities as well as basic movement-based activities, we also incorporate location information. Because we want to ensure that the model generalizes to multiple persons, we do not utilize specific location values. Instead, we construct person-invariant location features. This approach supports generalization without needing to employ techniques such as domain adaptation, which requires a large number of labeled training instances for the initial group of subjects [58–60]. One such person-invariant location feature is the location type, obtained through reverse geocoding using an OpenStreetMap. We group these location categories into residence, workplace, road, and other. Other location features are calculated for each sensor sequence and include the number of changes in orientation in the sequence, number of stops/starts, the sequence overall trajectory, and the distance traversed.

Attempting to perform real-time human activity recognition “in the wild” in naturalistic settings raises numerous challenges. One such challenge is collecting ground-truth labeled data. Users can be queried for their activities. However, these queries need to be answered in the moment to avoid retrospective error. Although active learning and semi-supervised learning can reduce the number of queries [61, 62], the resulting interactions will interrupt many of the activities we wish to monitor for our brain health study. To make the intervention as naturalistic as possible, we do not ask the subjects to provide labels for their daily activities. Instead, we train an activity recognition algorithm using past data collected from 388 different smartwatch wearers. These earlier volunteers provided labels for their current activity when prompted at random times. These prompts resulted in 59,610 labeled instances for 12 activity categories. These activity categories are reflective of routine behavior and include chores, eat, entertainment, errands, exercise, hobby, hygiene, relax, school, sleep, travel, and work.

Each 5-second sequence of sensor data is considered as a separate instance. Although shorter sequences have been used with success for ambulation recognition [63], we found that slightly longer durations are needed to capture context for recognizing complex activities in naturalistic settings. A feature vector is extracted from each 5-second sensor sequence. The feature vector consists of standard statistical operations applied to each of the sensor types [64]. The vector also contains person-invariant location features. We experimented with multiple types of supervised learning algorithms and found random forest to perform the best for these data. The random forest also generates a prediction for one sequence before the next sensor reading arrives, allowing the activity recognition to occur in real time. The random forest employs bootstrapping and is configured to contain 100 trees with a depth limit of 10. Because the training data are not uniformly distributed across the activity classes, we weight instances inversely proportionate to the relative size of the corresponding activity class. Applying threefold cross validation to our labeled data, activity recognition yielded an accuracy of 0.86 for the 12 activity classes. For the remainder of this article, we analyze sensor data that is labeled using a random forest model trained on all the labeled instances.

Once continuous wearable sensor data are collected and labeled with activity classes, we extract digital markers that describe the person's behavior patterns. Extracting features from longitudinal sensor data is popular for analyzing specific behavior parameters. As mentioned in Section 2, researchers typically extract indicators that are specific to a behavior of interest. Depending on the goal of the study, they may include amount of screen time, activity level, steps, incoming/outgoing phone calls and texts, number of steps, amount of screen time, and sleep. We are interested in characterizing a person's behavior routine. This is expressed as a set of digital markers that express how a person's time is spent on different activities, the locations a person frequents and the amount of time they spend there, and the traditional measures of overall movement types and durations. Obtaining digital behavior markers that express activity routines as well as direct sensor measurements of location and behavior are possible because we automatically label sensor sequences with activity labels, as described in Section 4.2. Thus, in addition to generating statistical summaries of raw sensor readings as listed in Figure 2 [65–68], we also extract markers that indicate how each subject divides their time among the activity categories and where they spend their time on a routine basis.

Table 3 summarizes all of these digital behavior markers that we extract and analyze. Our activity recognition algorithm applies statistical operators (listed in Figure 2) to raw sensor readings, yielding a feature vector that is mapped to an activity label. When assembling digital behavior markers, we apply these same statistical operators to aggregated daily and hourly sensor values, activity information, and location information. Additionally, we compute a regularity index to each variable as defined by Wang et al. [26]. The regularity index determines how repetitive a person's behavior routine is. This is computed as the difference between hourly statistics occurring during the same hours across two different days. To calculate this index, data are first scaled to the range [–0.5, 0.5]. The regularity index comparing days a and b is then defined as shown in Equation (1).

For our analysis, T is 24 hours and \(x_t^a\\) represents the value of a behavior marker for hour t of day a. We average the regularity index over all pairs of days occurring within the same week, all pairs of weekdays occurring within the same week, and between different weeks for the same day-of-the-week value. Finally, we compute the diurnal and circadian rhythms for each of the hourly values. We follow guidelines by Depner et al. [69] to define the circadian phase as constancy in the behavior routine. We quantify this definition by generating a periodogram for the marker values and calculating the normalized periodogram values for a 24-hour cycle (circadian) and for the 12 hours following the end of nighttime sleep (diurnal). Based on Fourier analysis, the spectral energy of each cycle frequency can be calculated as shown in Equation (2).

In Equation (2), N represents the number of samples in the time series; \({x_i}\) represents the value of the measured variable at time i; and the spectral energy is computed for a particular frequency, j. We calculate the spectral energy for a range of possible frequencies. The resulting periodogram generates values for a set of possible cycle lengths. The corresponding circadian rhythm parameter is then represented by the normalized periodogram-derived value for a 24-hour cycle, whereas the 12-hour post-sleep cycle is computed for the diurnal parameter. At the completion of the feature extraction process, our system has constructed a vector containing 102 behavior features for each observed day, 40 features for each observed hour, and 863 overall features for each person.

First, we verify the reliability of our computational methods that apply PCD-based change detection to digital behavior markers as a method of quantifying change in overall behavior routine. To verify that PCD computes change scores as expected for digital marker data, we examine scores that are generated for synthetic data. We use real digital markers from one randomly selected intervention subject. We then add white Gaussian noise with monotonically increasing values for the Gaussian distribution standard deviation. Table 4 verifies that the PCD-based change score increases monotonically with the amount of noise, or change, that is added to the real collected data.

Next, we analyze the collected sensor data to determine how much change occurs for one of the targeted activities. Our primary goal of this work is to determine whether our computational method can be used to detect and quantify change in overall behavior activity patterns as promoted by the intervention. However, we also posit that the same method can be used to detect change in a specific activity. The set of activities that are automatically recognized are listed in Section 2. The digital markers reflect these specific activities as well as parameters such as visited locations and movement statistics. Here, we pick a representative activity to demonstrate that the computational method can be used to analyze individual activities as needed.

We select exercise as the individual activity to analyze. One rationale for this choice is because the week discussing physical activity occurs early in the intervention. If behavior change is sustained per the intervention goal, change should be sustained throughout the data collection period in comparison with the week 0, pre-intervention baseline. A second motivation is that exercise represents a small set of possibilities for the participants (primarily walking and swimming). In contrast, the intervention topic that occurs the week before, cognitive engagement, is manifest through a large variety of individualized healthy behavior choices (e.g., play guitar, read a book, learn a new language), not all of which can be easily detected by smartwatch sensors. We include a subset of the digital markers for this analysis, namely the amount of time spent per day on the automatically labeled exercise activity, time of day that exercise occurs, the regularity of these activities on a daily basis, and statistical operations applied to these markers.

Table 5 summarizes the results of this analysis. The PCD values indicate that there is a greater change in exercise habits for the intervention group than for the control group. These changes are statistically significant for more than half of the group. Because PCD provides change amounts without indicating the nature of the change, we also compute linear regression for exercise time and note that the slope for the intervention group is slightly higher than for the non-intervention group. In fact, the coefficient for the non-intervention group is close to zero, indicating that the PCD-detected changes may be due in some cases to natural variation in exercise regimens rather than intentional increase in exercise amount.

The analysis of change in exercise markers confirms that some behavior changes can be detected by wearable sensors and that our PCD algorithm can quantify these changes. These findings align with prior work that detects changes in exercise patterns [72]. However, we are particularly interested in observing the impact of a behavioral intervention on a person's entire pattern of behavior. Researchers have discovered interplay between many behaviors, including but not limited to exercise and work [73], work, travel time, and social life [74], and diet and sleep [75]. Therefore, we hypothesize that engagement in the BHI will impact overall behavior patterns. We analyze the complete set of digital markers collected for study participants and for the non-intervention group and compute PCD scores that compare the baseline week with the end-of-collection week.

The results of this analysis appear in Table 6. As anticipated, the changes are greater for the intervention group than for the control group. However, weekly changes were still fairly high for the non-intervention group. When we plot weekly change scores for individuals within the two groups, we gain new insights on the nature of the change. Almost all of the individuals in the intervention group exhibited consistently increasing change from baseline, as shown in Figure 3 (left). In contrast, the trends were highly varied for the non-intervention group. Figure 3 (right) provides one such example illustrating a common theme among non-intervention participants. To quantify these trends, we compute the linear regression coefficient for the change scores.

Next, we examine the widespread impact of a BHI on a person's entire daily routine by assessing change in the types of locations where a person spends their time. To visualize the impact that intentional change can have on location patterns, we plotted the normalized, aggregated time spent in the home and outside the home (work, socializing, hobbies, or exercising) the week before intervention initiation and the last week of the intervention. Figure 4 shows this visualization for a 24-hour timeframe. As the graph shows, end-of-intervention routines include more time spent at home in the late-night and early-morning hours. One possible explanation for this is a change in behavior to incorporate more regular sleep hours and patterns. Furthermore, the midday hours reveal less time spent in travel (driving or riding other public transportation) and more time spent in other locations out of the home. This could potentially be attributed to incorporating more exercise, social times, and hobbies into the person's daily routine. Change in location markers between 0 and 9 were much greater for the intervention group, averaging 0.8469, whereas change for the non-intervention group averaged 0.3279.

In our final analysis, we employ a supervised learning algorithm to classify each subject into their corresponding group (intervention or control) based on digital behavior markers. Because we are focused on behavior changes related to the BHI, we compute delta features to perform the classification. Specifically, we calculate the numeric change in each marker from week 0 to week 9. The set of delta features is mapped to the predicted subject group using a range of alternative supervised learning algorithms. To quantify the impact of the intervention on a person's overall activity routine, we compare the results using all delta features with those derived from using a subset of the available features. Specifically, we compute predictive accuracy using only delta features derived from movement-based sensors, labeled activity features, or location type features.

The results of this analysis are summarized in Table 7. We utilize a random forest with 100 trees and a varied number of features. Results are computed using leave-one-subject-out validation. As the table reflects, the delta values (calculated between the first and last weeks of data collection) were predictive of the subject group with an accuracy of 0.87 when all the features were included in the learned model. The ROC curve for this problem is shown in Figure 5. As the figure indicates, the corresponding AUC (area under the receiver operating characteristic curve) is 0.95.

Using subsets of features still resulted in an accuracy that was better than random chance. However, considering changes in only movement features, activity features, or location features did not produce as strong results as considering changes in the entire set of features, corresponding to the person's larger behavior routine context.