Voice Assistive Technology for Activities of Daily Living: Developing an Alexa Telehealth Training for Adults with Cognitive-Communication Disorders

Prior studies have reported VAT as easy-to-use and accessible tools for individuals with cognitive communication impairments when engaging in their daily activities [8, 16, 18-22, 48]. Goodwin and Brandtzaeg [49] analyzed the perceived benefits and challenges of using conversational user interfaces (CUI) with older adults with cognitive impairments from neurologists and web developers using semi-structured interviews. Using CUI enabled more engagement with human-like social interaction. Kulkarni and colleagues [24] described the use of Google Home among adults with aphasia via speech-to-text (57%), speech therapy practice (43%) and ADLs (28%) with instructions from a speech-language pathologist (SLP) in the clinical setting. Using Google Home helps to simulate “real life” settings for information searching, functional writing (e.g., emails and texting), and performance of ADLs, leading to increased independence, sociability, and confidence. Other studies reported older adults benefited from the use of VAT for increased activities of daily living, gaming for entertainment, and online connection with others [21, 22, 28, 43]. Prior work also found that older individuals with cognitive and visual impairments were found to show increased memory and task completion [18] and use of voice commands using VAT when provided compensatory strategies to perform functional tasks [19, 20].

On the contrary, some studies found challenges for these individuals with CCDs when using VAT [17, 48, 52]. Although many studies have explored the use of VAT among users with physical and visual impairments [22,33], due to a wide range of factors (e.g., different etiologies), individuals with cognitive and communication impairments may have complex communication needs that hinder their ability to use VAT [7,17,24]. While some studies suggested VATs such as Google Home and Amazon Echo were able to pick up most commands produced by individuals with TBI regardless of their cognitive and linguistic abilities [16], others reported critical usability challenges (e.g., pronunciation and phrasing errors due to speech recognition and natural language processing, timing errors due to response latency) [7] and accessibility issues (e.g., set up process, stable internet connectivity) [15]. For instance, to interact with VAT, users must demonstrate an adequate level of speech and cognitive skills [7]; to engage in extended conversational exchanges, individuals should have the ability to interact with the device in a timely manner, remember specific keywords, and utter concise command sequences involving planning, executive functioning, attention, long-term memory and working memory [8]. Therefore, accessibility and usability be significantly affected for individuals with CCDs, and additional trainings and supports are needed to support these individuals.

The existing clinical intervention for individuals with CCDs predominantly revolves around utilizing mobile apps, alarms, digital calendars, and remote patient monitoring systems to serve as memory aids [38]. Oyesanya et al. [23] found that individuals with TBI who return home from medical settings have difficulty regaining their cognitive functions; however, the current literature has not thoroughly explored the potential adoption and feasibility of employing VAT within the home environment to compensate for cognitive needs such as memory, attention, and executive functioning. In an exploratory study with persons with TBI and their caregivers on the use of technology to address health, wellness, and safety concerns, [24] reported that the persons with TBI and caregivers wished to address technological solutions related to medication management, mobility, cognitive functioning, and social relationships. Based on the known affordances of VAT, it is possible that using VAT to help individuals with CCDs to not only improve their comprehension skills and ability to produce clear sentences but also execute functional daily tasks, which is also a key emphasis for communicative-cognitive rehabilitation [4-6]. Such training on communication and cognition, in turn, led to an increased independence and self-esteem and more opportunities to return to work for continued occupation [25]. These findings are a strong rationale to develop clinically informed training around ADLs skills for individuals with CCDs.

While VAT devices are becoming increasingly ubiquitous, their actual usage among adults with CCDs remains low. Existing literature has shown that within the older adult population, many individuals have felt that technology has not developed to cater to their specific needs and levels of technology literacy [12, 25, 26]. Studies on the interaction between VAT and its utilization by older adults with CCDs remain relatively sparse, and very few investigated VAT use in the natural home environment for users [8, 12]. For example, [12] reported that older adults primarily used digital sources for objective health information but did not find voice assistants (VAs) such as Amazon Alexa to be adequately tailored to their medical requirements. While [19] and [25] studied the effects of VAT use by adults with cognitive-communication impairments, a limitation arises in the fact that their studies failed to mimic a natural home environment due to control of noise level, distractions, and group interactions (e.g., multiple speakers). By 2050, it is estimated that 22% of the world's population will be over 60 years old (approximately 2 billion older adults) [45]. There is significant room for better design and implementation of VAT to fit the daily, functional needs of the growing older adult population among community-dwelling adults with CCDs [12,18].

This study encompassed two rounds of training sessions via two cohorts with a total of seven individuals with CCDs, in order to ensure quality of training for these individuals in a group telehealth service delivery format. Table 1 presents the demographic details and attendance records for the participants in the VAT training sessions. Cohort One (P1, P2 and P3) participated in the study for a total of 6 weeks; Cohort Two (P4, P5, P6 and P7) participated in the study for a total of 8 weeks, with the two extra weeks added to give participants additional breaks and times to complete pre-training baseline measures (Week 1) and post-training debrief interview (Week 8). Recruitment for both cohorts was conducted through word-of-mouth referrals and social media (e.g., Facebook Groups for TBI). To be eligible for the study, participants must (1) be adults with neurological impairments aged 18 years or older, (2) primarily residing at home with a minimum of 1-1.5 years post-injury without reported difficulties with vision or hearing, (3) possess stable internet and computer access with a functioning web camera, (4) receive adequate home support (defined as having at least one caregiver available for technology setup and troubleshooting), (5) be capable of completing all training via video conferencing and (6) have no prior VAT training on ADLs prior to the study. Across both cohorts, a total of 12 participants were initially recruited and only seven individuals completed all the pre-training testing for enrolling in the study. All participants received an Amazon Echo Show 5 device for use during the study period and as study compensation to keep after the study completion. This device contains a 5.5” screen, providing multimodal output to ensure increased accessibility to its users. Caregivers for P1, P3 and P5 were present during the VAT training. P3’s caregiver actively participated in each session, while the caregivers of P1 and P5 were present in the room but did not participate in the training. Cohort One primarily consisted of individuals with mild cognitive impairment resulting from TBI, spanning a more diverse age range (see Table 1). Conversely, participants in Cohort Two exhibited various disorders (e.g., TBI and aphasia) but generally fell within a similar age range.

Participants who attended all to almost all sessions demonstrated varying consistencies between 17% and 100% with minimal to maximal cueing, showing gradual improvement in the execution of commands, as summarized in Table 5. While P1 and P2 required assistance when finding specific information (i.e., a certain movie trailer), P4 stated that she required assistance when asking Alexa for a doctor nearby (i.e. a medical doctor). Due to difficulties with working and short-term memory, P1, P4, P5 and P7 expressed the most interest in using Alexa to help simplify the process of carrying out ADLs by having Alexa remind them of tasks that needed to be completed such as “taking medications, putting a load [of laundry] in and hanging towels in the bathroom.” When asked what they used prior to using VAT for their reminders, P4 and P7 had mentioned “writing down what they needed to do” on a piece of paper. P7 reported on relying on his wife to remind him to write down the task first for him to remember it later. In addition to using Alexa for reminders, P5 and P2 had expressed interest in utilizing more of Alexa's functions independently such as “asking for the weather”, asking for a specific doctor and playing videos and music.

Throughout the training, each participant had their own levels of proficiency when executing commands. P1 and P2 had maintained consistent accuracy for each week, P1 scoring overall accuracy of 92% with minimal to no cueing and P2 scoring 100% accuracy with minimal to no cueing. P4 and P5 had also maintained similar consistencies ranging from 75-76% accuracy with P5 requiring moderate visual and verbal cues when including essential verbiage to allow for better identification of context. For instance, P5 would require a verbal cue to include the appropriate time of day (i.e., am and pm) desired hour of the day, “Alexa, remind me at 7 o'clock to shower every evening.” P7 was observed to show the most challenge and frustration towards device functions, indicating that it “doesn't hear what he is saying.” He required maximum verbal and visual cues, showing an overall accuracy of only 40%. Various challenges were overcome through clinician prompting and instruction, who guided participants to troubleshoot the device or use of intelligibility strategies such as slowing down, increasing volume, over-articulation, and condensing long utterances. Despite challenges experienced, all participants had expressed interest in pursuing more of Alexa's functions during their final debrief interview. Furthermore, cognitive linguistic skills in the areas of memory and attention were targeted via various strategies embedded within the training sessions. Memory was targeted through introduction of memory strategies such as setting alarms/reminders and repetition of commands to enforce recall. Sustained and alternating attention skills were targeted via aspects of the training that involved sequencing steps to troubleshoot the device. To reduce the length of verbose commands, clinicians administered modeling and presentation of concise commands written on the screen throughout both training sessions. These strategies were effective in guiding patients to execute appropriate commands for Alexa. 

In Week One of training, participants received a brief introduction to the curriculum, facilitating collaboration between clinicians and participants to establish rapport and familiarity with devices. Participants were informed of skills to cover each week, time commitment and anticipated training outcomes. Following the introduction, the first lesson focused on scheduling events and setting reminders. Clinicians demonstrated setting a reminder for the training at 5:30pm every Thursday. Participants were then provided with a step-by-step “roadmap” for executing voice commands. It was essential for participants to be well-informed about the prompts for setting events or reminders. The roadmap served as a visual guide to help participants break down commands for scheduling. Figure 1 illustrates a notable divergence between participants' self-reported baseline ADL measures and their accuracy in executing voice commands during Week One of VAT training. The data suggests a potential inconsistency, with some participants reporting high ADL measures but demonstrating lower accuracy in voice command execution. This discrepancy may be attributed to participants still acclimating to the device, leading to larger gaps between self-reported capabilities and actual performance.

During Week Two's training session, clinicians aimed to teach commands for navigating entertainment platforms, introducing participants to commands that would automatically open YouTube for sourcing entertainment and media. Using a visual roadmap, the session encompassed instruction on-device controls, such as the pause, volume up/down, fast forward/go back, play from the beginning/restart again and the go-home features. Figure 2 compares self-reported ADL measures and actual accuracy during Week 2, revealing disparities for some participants, like P1 and P5. The table provides insights into the alignment or variance between perceived ADL capabilities and actual performance in voice command execution.

Training in Week Three focused on self-care, hygiene, and medical needs, spanning two weeks for Cohort Two participants due to time conflicts. The clinician commenced by providing examples of self-care and hygiene, emphasizing Alexa's ability to assist in setting reminders for crucial appointments, locating COVID test and flu shot locations, and searching for nearby doctors, as well as hair and skin necessities. Additionally, the clinician showcased the touchscreen feature of Alexa, demonstrating to all participants how they could tap on the device and make selections using their fingers. Figure 3 provides a comparison between participants' self-reported baseline ADL measures and their accuracy in executing voice commands during Week Three of VAT training. Notably, participants like P1 reported perfect self-assessment in ADL, which aligned to his actual percent accuracy achieved. Conversely, participants like P3 self-reported a higher percentage in ADL but attained an accuracy of 44% accuracy, highlighting a potential inconsistency. The table offers insights into the concordance or variance between participants' perceived ADL capabilities and their actual performance in utilizing voice commands during the third week of training.

Week Four of training highlighted commands on News, Facts and Communication. P2, P3, P4, P5 and P7 were in attendance for this week's training, while P1 and P6 were not present. Participants encountered challenges related to device responsiveness and recognition, occasionally requiring device restarts to restore functionality. Clinicians addressed these difficulties by encouraging participants to say “Alexa, go home” or “Alexa, stop” whenever a breakdown occurred or if the device did not pick up their commands accurately. Figure 4 illustrates the comparison between self-reported baseline ADL measures and accuracy in executing voice commands during Week Four of VAT training, focusing on "News, Facts, and Communication." Notably, absent participants P1, P4, and P6 are indicated with dotted bars, while present participants, such as P2, exhibited a high correlation between reported ADL measures and accuracy in executing voice commands.

Week Five's training module consisted of commands for utilizing Alexa to assist with meal preparation (e.g., finding recipes, creating a shopping list, inquiring about nutritional facts and food substitutions). Figure 5 depicts the comparison between self-reported baseline ADL measures and accuracy in executing voice commands during Week Five of VAT training, focused on "News, Facts, and Communication." Participants P2 and P6 were absent from the training, and their missing data is represented with dotted bars in the graph. Noteworthy observations included P1 reporting a 50% ADL measure but achieving 100% accuracy, and P7 reporting an ADL measure of 96% but achieving an accuracy rate of 44%.

This virtual training offered a platform for participants to acquire new skills within a communal/group environment. Participants were able to gain insights from one another's experiences with the Alexa device and collaboratively engage and problem-solve on one another's successes and challenges. Feedback gathered from debrief interviews from participants and caregivers reported perceived benefits and utility of this VAT training led to positive perceptions from participants. VAT for adults with CCDs demonstrated the use of VAT to increase independence in ADLs and information seeking, which were consistent with prior studies [24,43]. Since P5 had difficulty reading, written cues were not used to help elicit working commands. Due to personal dietary restrictions, P5 expressed satisfaction after learning that she could ask Alexa whether the foods she ate regularly were relatively “healthy” or “unhealthy” for diabetes. P5 engaged with commands encompassing weather, scheduling, reminders, and communication, demonstrating the ability to rephrase commands to suit her communication style was evident, and portraying confidence in tailoring commands to her requirements. P5’s engagement was particularly visible in the commands related to news, facts, and communication. Additionally, she adeptly used prompts to halt Alexa and effectively manage the creation, modification and cancellation of reminders and appointments. VAT also consolidated needs for P6 who inquired whether Alexa could create a specific timer (e.g., a boiling egg timer for 11 minutes). In sum, the ability to independently execute these voice commands allowed participants to improve independence and generalization of ADL skills through VAT, as noted in prior studies [24,43]. Furthermore, caregivers who were present throughout the VAT training additionally reported perceived benefits regarding participants’ increased independence at home. From the perspective of caregivers, [29] states that there are 50 million unpaid family caregivers in the U.S., which amounts to a $600 billion segment of people performing roles that they were not trained for using VAT [30]. One of the primary challenges faced by caregivers is their lack of awareness regarding how technology can assist them and their loved one at home [30]. By allowing caregivers to be part of the training and debrief sessions, caregivers gained knowledge and skills regarding using VAT to assist their loved ones to be more independent with ADLs.

Some user accessibility challenges included difficulty with speech recognition while using Alexa and where a common complaint during the debrief interview was related to “Alexa cutting speech off.” This would be challenging particularly due to the participants’ language and cognitive communication deficits. Consistent with prior research [8], to converse with VAT, participants need to use specific words in a timely manner and to produce appropriate sequences of commands that need planning, execution and working memory. Additionally, clinicians had also expressed concern regarding competency in accessibility with the device and the application that is used to set up their Alexa account. The clinician who had worked with P7 had stated that Alexa would “cut him off” before he finished the command. Despite challenges experienced with P7, the clinician was unaware of how to delay Alexa's response time and agreed upon further education regarding accessibility functions of Alexa. Before learning skills using Alexa, learning how to set up the device using the Alexa app is a crucial part of this training. Creating a diverse device environment by teaching participants and clinicians how they can navigate the Alexa app would allow for independent troubleshooting and better device management. These changes are expected to enhance participant engagement and improve the effectiveness of the training.

Customized training strategies were incorporated for some participants to learn using the VA considering their special needs and/or deficits due to comorbidities. Addressing low literacy with P5 involved reducing reliance on visual cues during therapy sessions. Instead, instructions were delivered through various modalities, including verbal models. Before this training, P5’s ability to search the same information by using graphic user interface (GUI) was very difficult for this participant due to her lack of literacy skills added on to her cognitive-communication difficulties. Difficulties with GUI were likely due to her lack of literacy skills added on to her cognitive-communication disorders, and such challenge was also reported in individuals with visual impairments [13]. With multimodal supports and prompting from clinicians, P5 demonstrated consistent cooperation throughout the session, achieving a 97% accuracy in command execution despite requiring minimal cues. Similarly, due to P7’s diagnoses of both TBI and aphasia, we provided multimodal stimulation consisting of spoken and written commands for P7, where we attempted to reduce the visual distractions from the screen and wrote the commands in a larger font size using the whiteboard feature on Zoom or on a Google doc for them to repeat these commands without getting confused once the wake word was spoken for each command. Additionally, Alexa was deactivated to facilitate verbal repetition practice. By piloting the study via telehealth with participants in their most naturalistic environments, we not only piloted the feasibility of offering easy access to such services but also identified several accessibility considerations through the training (e.g., low literacy, low-vision, instruction being provided with various modalities, etc.).