Collaborative School Mental Health System: Leveraging a Conversational Agent for Enhancing Children's Executive Function

Executive function is defined as self-directed behavior, including goal selection and prolonged self-control towards the achievement of goals, which is widely covered not only in the clinical field but also in the field of child development [48, 33]. Executive function enables children to maintain their level of concentration, retain pertinent objectives, resist impulses, endure frustration they might face, weigh the outcomes of their actions, and be prepared for the future [42]. This aligns with the discovery that children with strong executive function often exhibit enhanced academic performance [41]. However, beyond academic performance, executive function is generally considered a vital ability to possess for people to successfully achieve goals in various situations they may encounter in the course of their lives [50].

Studies using brain imaging methods have shown that executive function is deeply related to the frontal lobe and continues to develop and change until adolescence [50, 30]. Executive function is known to develop most rapidly from early childhood through the lower grades of elementary school, although the timing of development in each area can vary [10, 8].

To improve executive function, it is important to allow children to develop their own routines that are made out of their autonomous decisions and for them to get used to those routines. First, children should be able to control their daily lives willingly. A study of children aged 6–7 years showed that children had higher self-directed executive functioning when they performed activities that they chose to do compared to when instructed by their parents [23]. Therefore, children must experience the entire process of managing and performing activities based on their decisions. Second, children should become accustomed to it through repetition. High levels of practice, repetition, child engagement, and real-world generalization are considered some of the most important factors in interventions aimed at improving executive function [2].

Therefore, in this study, a prototype was developed to enable elementary school children to independently set their own goal behaviors, perform these behaviors, and evaluate the outcomes. The aim was to promote the enhancement of children's executive function through interaction with this prototype.

ADHD prevalence is known to be 5–7% worldwide [20]. However, in a study using Korea's National Health Insurance database, less than 1% of children were diagnosed or treated, confirming that less than 1/5 were receiving treatment [40].

Schools can play a valuable role in the screening and treatment of ADHD. First, teachers can suspect that a student is an ADHD patient more quickly than anyone else because ADHD symptoms can differ somewhat from those of typically developing children [44]. Teachers work with many children and experience various cases [4]. Therefore, teachers can play a significant role in screening children for ADHD, thereby leading to early treatment.

Second, schools can also play a crucial role in the treatment of children with ADHD. In the United States, medication and school support are identified as treatments applied to 2/3 children [38]. In this study, 66.9% answered that they were taking medication and 64.7% were receiving school support. In particular, in the United States public education system, Individualized Education Programs (IEPs), or 504 Plans, are provided to children with ADHD, and 51.6% of them receive such education [12]. Such programs have made significant contributions to ADHD treatment.

Third, the primary benefit of school mental health programs is that it increases student participants’ accessibility and reduces the chance that they will feel attacked by negative stereotypes [9, 31]. Despite significant improvements in awareness of mental illness, there are still fears that other people are judgmental and biased towards certain mental health issues [5]. Having a relevant program as part of universal school practice is expected to reduce the level of concern.

Thus, school-based support systems could play a significant role in the diagnosis and treatment of ADHD. Furthermore, schools can have a significant impact on children with ADHD, as well as on those with broader mental health concerns, throughout the diagnostic and treatment process.

Therefore, this study applied a research design that considered children who may have ADHD but have not yet been diagnosed. Additionally, the intervention applied in this study, which targeted children with ADHD, also demonstrated benefits for typically developing children.

This study is based on voicebot interaction research previously conducted as a tangible product targeting children with ADHD [13, 14]. The voicebot had a physical presence that increased its effectiveness in children with ADHD. However, its limited versatility made it challenging to conduct experiments with a large number of participants. In this study, we integrated a chatbot builder, NAVER CLOVA, into an Android application that is compatible with tablet PCs to develop a conversational agent. In other words, we redeveloped an application that allows voice interaction. This prototype is an application that helps children start and complete their daily tasks, which they set up with their parents (Figure 1 (a)). The daily task list was designed to address the overall daily tasks of children in lower elementary school grades. There are a total of 18 daily tasks, including waking up in the morning, brushing teeth after dinner, preparing bags, and reading books.

In previous studies, self-instruction and behavioral parenting training have been incorporated into the interaction process to assist children with ADHD from a cognitive-behavioral therapy perspective [13]. Cognitive-behavioral therapy enables children to develop strategies for monitoring and managing their own executive function [11]. In particular, self-instruction involves acquiring strategies for self-regulation, which has been found to be effective in regulating behavior in children's daily lives and learning domains [58]. Behavioral parenting training is widely recognized as one of the most effective interventions [54]. An exemplary case involves the use of tokens to reinforce positive behaviors and reduce negative behaviors in children [37].

The application developed for this research was designed using a “Goal, Plan, Do, and Check” process, primarily based on self-instructional steps [16, 21]. This process enables children to 1) set their own task goals, 2) receive reminders about these goals, 3) implement them, and 4) confirm their progress. Rewards were provided to children through a token economy system. When a child completed their daily tasks, they received a star sticker, and upon successfully completing all tasks for the day, the star sticker was replaced with a character sticker. In addition, a character card was also given (Figure 1 (b) and (c)).

When impaired executive function impairs the process of regulating behavior through internally represented forms of information, obtaining externalized forms of information is helpful [46, 47]. Notably, the physical representation of information must be externalized when performing a task [47]. Therefore, we determined that voice interaction might be necessary for children with ADHD. The prototype tablet PC application delivers both visual and auditory cues. This prototype uses cues to convey information about a child's situation.

A conversational agent, which has widely used functions such as ordering goods and weather guidance, is structured to appropriately answer its user's question [60, 64]. In our prototype, the typical relationship between the user and the chatbot is reversed. This means that ForME was designed to communicate with its users by prompting them to respond with several command words with limited intent, so that conversations can be conducted in a rule-based manner. The core interaction of this prototype enables child users to individually perform several actions one after the other.

Children would normally deliver a speech with four main intentions [14]. In some situations, dialogue may be determined by three or five different intents, depending on the context. First, there is the correct answer section, where the child has completed a certain task successfully and proceeds to the next task. When the current activity has not been performed completely or is performed incorrectly, the second section is defined as in-progress. In the third section, the unable-to-process section, ForME is unable to process the child's response because of a lack of clarity in the child's speech or accent. The final section, or the no-response section, occurs when there is no response from the child at all. The children's responses were organized such that the prototype could maintain communication with them based on their responses.

These tasks can be performed in two ways. First, the agent ForME provides a sound alarm at a preset time, saying "What should we be doing right now?", so that the child can fulfill their set goal. Second, the child can start fulfilling the goal by tapping the ForME icon in the application.

The cue was designed with reference to smart speakers and displays. Figure 2 shows a child and the ForME agent working together to complete the task of “Brushing teeth after dinner” when the prototype alarm goes off at the pre-arranged time. The agent delivers a cue to the user through two channels: sounds and visuals. The sounds consist of two components: a beep that serves as a cue to help participants better understand the contents of the conversation and a dialog that contains the actual conversation. There are four different beep sounds: ringing, ding, tick tock, and ding-dang-dong. Ringing indicates the start and end of the entire task. Ding signals when users start and stop talking. Tick tock indicates the start of the timer, and ding-dang-dong indicates the end.

For each phase, multiple paraphrased versions of the dialogues are generated, totaling five or more alternatives for a given scenario. One of these alternatives is randomly chosen and presented to the user. Visual cues accompany each step to illustrate screen modifications. Adjustments in the steps and the agent's expressions reflect advancements in the task. Figure 2 illustrates children's responses and actions during their interactions with the ForME agent.

When the user does not mention the required keywords or does not speak at all during each step, the agent asks for a response so that the user can provide an answer that includes the keywords. For example, when the agent asks the user which task should be performed (see Figure 2) and the user provides a wrong answer, such as “wash my hands” instead of “brush my teeth,” the agent produces an output that says “You know…there's something you have to do three times a day to avoid an ache! Tell me what that is!”, which allows the user to derive the correct answer via the hint given. The keywords that the agent perceives as the correct answer at each step were developed using the Natural Language Understanding function provided by the chatbot engine, which was developed to analyze and process user intentions.

We configured the prepared scenarios using the NAVER CLOVA chatbot builder, which is equipped with natural language processing technology for Korean as well as a built-in machine learning algorithm. The Speech-to-Text functionality embedded in Android was prioritized. However, in cases where the analysis of a child's speech is ambiguous and intent recognition is challenging, we devised a system to verify using CLOVA Speech-to-Text. The server was divided into a Web App and API Server. The Web-based app is deployed to the client using the service through Google Firebase Hosting, and it processes service requests such as conversations and appointment schedule checks from the client and returns the results. Because the CLOVA Chatbot Builder does not provide built-in responses for waiting for appointment times or handling no-response intent, we implemented Chatbot Middleware to enable these functionalities.

We developed a Unity-based mobile application for parents to monitor their children's use of ForME. Thus, parents can install the parent application regardless of the operating system of their smartphone.

Parents and children can use the parent application to register their child's information, input school details, and enter the corresponding school code, as illustrated in Figure 1(d). Subsequently, as depicted in Figure 1(e), they can obtain a QR code, which the child can use to log in to the child application.

Parents can track their children's daily task performance and view the tasks they have completed through a dedicated web page for parents (Figure 1(f) and (g)). It is important to note that updates on task performance are not in real time but become accessible after a day. Real-time alerts and notifications were not incorporated to promote the children's autonomy and to avoid excessive intervention. Instead, parents were encouraged to agree with their children before using ForME, allowing the child to maintain a high degree of autonomy. However, if difficulties arose, the parents were informed that they could assist their children.

We developed a webpage that allows homeroom teachers at the school to manage each class's students who participated in the research. Homeroom teachers can issue codes for their respective classes and distribute them to link student accounts to their own.

As depicted in Figure 1(h), each homeroom teacher can access and review their students and monitor how they perform daily tasks. School administrators and counselors can also access information on students in each class and check their task performance rates.

As a result, each teacher is able to individually manage the daily lives of the participating children at their homes. Counselor teachers also have the ability to gain insight into how the children are living at home, making it possible to use this information for counseling purposes.

This study received institutional review board approval. This study was conducted on children in their first year of elementary school. The schools were recruited through a metropolitan city education office in Korea, and 20 schools participated in the study. Considering the number of students attending each school, two schools were paired, and using a random number table, each pair was randomly assigned to the intervention and control groups. As a result, 10 schools were assigned to the intervention group and the other 10 to the control group. The total number of students from the 10 schools assigned to the intervention group was 1,046 and the control group was 929.

Letters promoting participant recruitment were distributed by each school to each household. The application for study participation was collected with the consent from the students and their guardians. The researchers then had a call with each parent to explain the study in detail. Subsequently, the students and their parents provided consent to participate in the study through electronic signatures within mobile messages, and each student was given a screening number. The list of participants was finalized by the researchers after confirming whether the student had 1) mental diseases other than ADHD and 2) registered for other clinical studies.

The first questionnaire for parents was delivered to each household by mail. Once the parents completed the questionnaire, they were asked to take a picture of the document and return it to the researcher via phone. The researchers checked for missing items and collected the survey data. The missing survey items were requested from the parents again. Questionnaires for teachers about each student were distributed and collected through e-mail. The researchers once again checked for missing information and processed the survey data.

Participants in both the intervention and control groups completed their first questionnaires in May. After completing the survey, the intervention group was instructed to use ForME for the next six weeks by entering a code that they were given. The second questionnaire was administered in June. After completion of the survey, the control group had the option to use ForME for six weeks if they wished.

After completing all phases, we collected feedback from parents and teachers about the program through an online survey.

A total of 260 participants applied for the study through the recruitment letter sent out from each school. The researchers contacted the children's parents to confirm the necessary information, resulting in a final count of eligible participants who registered as subjects. There were several reasons why some of them had to eventually give up registering, which included not having enough time to conduct two surveys, because it was not a mandatory program assigned by the school, and not having an Android device at home. Ultimately, 179 participants completed the first survey, among which, 132 completed both surveys. Figure 3 presents more detailed information regarding this.

Initially, an independent t-test or Mann-Whitney U-test was conducted to assess homogeneity based on the general characteristics and initial outcome results of both groups. To facilitate this, the normality of the data was assessed using Kolmogorov-Smirnov or Shapiro-Wilk tests. Second, Cronbach's alpha for each factor was verified. The reliability of an assessment is considered acceptable when Cronbach's alpha exceeds 0.7 [28]. Depending on the normality of the data results, either Repeated Measures Analysis of Variance or Generalized Estimating Equation (GEE) was used to analyze the differences between the two groups at two time points with a 6-week interval. That is, when the data did not follow a normal distribution, it was analyzed using GEE, and the family of models was selected based on the characteristics of the data. Lastly, we examined whether there were changes in executive function and ADHD symptoms in children classified as ADHD-suspected in the intervention group. Depending on the normality test results, the data were analyzed using either a paired t-test or the Wilcoxon signed ranks test. Prior research indicated that children were considered potential ADHD cases if their total ADHD RS score, as assessed by their parents, was 19 or higher, and if their teachers’ assessment yielded a score of 17 or higher [22]. Categorization of the ADHD-suspected group was determined based on the ADHD RS scores evaluated by the homeroom teacher. Statistical analyses were conducted using SPSS 26.0, and a p-value < 0.05 was considered statistically significant. The feedback received from parents and teachers was analyzed through qualitative summaries.

The task completion rates were checked to confirm whether the children in the intervention group actually used the prototype for the 6 weeks. Children are able to perform tasks they agreed to conduct with ForME, which are 1 to 4 tasks per day. Each child's task completion rate was analyzed on a daily basis. Consequently, the 67 children assigned to the intervention group demonstrated an average task completion rate of 56.2% per day. Additionally, more than one task per day was performed for an average of 31.8 days during the 42-day ForME trial period.

All participants whose data were analyzed were first-year elementary school students. Of the 67 participants in the intervention group, 34 were male and 33 were female (Table 1). There were 28 6-year-olds and 39 7-year-olds. Of the 65 control group students, 38 were male and 27 were female, of which 23 were 6-year-olds and 42 were 7-year-olds. Participants in the intervention and control groups showed no statistically significant differences in terms of their demographic variables, as shown in Table 1.

When the normality test on the initial outcome was conducted using the Kolmogorov-Smirnov and Shapiro-Wilk tests, the normality assumption was not satisfied. Therefore, the Mann-Whitney U-test was performed on the intervention and control groups to assess homogeneity. Table 2 shows the homogeneity of outcomes between the two groups (p > 0.05).

First, the Cronbach's alpha values for each of the sub-factors in the outcomes exceeded 0.7, indicating their reliability (BRIEF: BRI = 0.896, MI = 0.933, GEC = 0.943; ADHD RS (parents): Inattention = 0.836, Hyperactivity/Impulsive = 0.770, Total ADHD RS = 0.883, ADHD RS (teacher): Inattention = 0.962, Hyperactivity/Impulsive = 0.950; Total ADHD RS = 0.974). Then, Kolmogorov-Smirnov and Shapiro-Wilk tests were conducted; however, the dependent variables in this study did not meet the normality assumption. Therefore, GEE was used to explore the effects of executive function and ADHD symptoms. In particular, it was analyzed as gamma GEE with log link, considering the characteristics of the data corresponding to continuous variables and being skewed to the left. The detailed statistical analysis results are presented in Table 3. The results are described with a focus on the interaction effect between group and time on outcomes. The interaction effect provides important information, suggesting that the differences between the two groups changed over time.

Initially, the interpretation of the sub-factors of executive function was as follows. GEE analyses indicated a significant group x time interaction for BRI (β = 0.052; Wald χ²₁ = 6.463; p = 0.011) and GEC (β = 0.044; Wald χ²₁ = 4.427; p = 0.035) but not for MI (β = 0.035; Wald χ²₁ = 2.736; p = 0.098). In other words, the use of ForME helped improve the BRI and GEC in children's executive function. Group differences in the patterns of change over time are shown in Figure 4. In the intervention group, scores for the sub-factors of executive function generally decreased, while in the control group, an overall increase in scores can be observed.

Next, interpretation of the ADHD RS as assessed by the parents was as follows. GEE analyses indicated a significant group x time interaction for Inattention (β = 0.346; Wald χ²₁ = 5.414; p = 0.020) but not for Hyperactivity/Impulsive (β = 0.300; Wald χ²₁ = 2.782; p = 0.095) and Total ADHD RS (β = 0.250; Wald χ²₁ = 3.181; p = 0.075). Among the ADHD symptoms evaluated by parents, Inattention indicates that the use of ForME significantly improved the child's symptoms. Differences in group patterns of change over time are shown in Figure 5. In the intervention group, there were no significant variations in the three sub-factors. In fact, the mean value of the Total ADHD RS did not change over the course of 6 weeks. In contrast, in the control group, the sub-factors increased over time, indicating worsening of symptoms.

The interpretation of the ADHD RS as assessed by teachers was as follows. GEE analyses indicated a significant group x time interaction for Inattention (β = 0.394; Wald χ²₁ = 11.368; p < 0.001), Hyperactivity/Impulsive (β = 0.286; Wald χ²₁ = 5.668; p = 0.017), and Total ADHD RS (β = 0.378; Wald χ²₁ = 10.651; p = 0.001). Among the ADHD symptoms evaluated by teachers, ForME usage significantly improved the respective symptoms in both Inattention and Hyperactivity/Impulsivity in children. Figure 6 illustrates the differences between the intervention and control groups. In the intervention group, the values of the sub-factors decreased over time, indicating an improvement in symptoms. Conversely, the control group showed an increase in factor values over time.

After summarizing the online feedback from both parents and teachers, we were able to gauge the effectiveness of the ForME program conducted over six weeks.

First, parents were deeply impressed by their children's efforts to control their behavior. Comments such as “Even if our son didn't do well, it was heartwarming to see him make an effort” and “I felt that my daughter has grown a bit while performing daily tasks with ForME characters” were received. Children's attempts to engage with ForME and perform daily tasks were seen as fresh and challenging.

However, there were instances in which some children lacked time or forgot about task completion, resulting in less meticulous task execution. Parents commented, “Our son doesn't have enough time for brushing his teeth for 3 minutes. I wish a little more time was allocated” and “My child attempted to pack the bag but found it difficult, so I, as a mother, helped.” Nonetheless, the parents demonstrated patience and provided support to help their children perform tasks independently.

However, in some cases, the program's utility was not evident. Some parents expressed, “Our daughter already had good daily habits, so she didn't need this program” and “I'm not sure how our child benefits from this program.” We interpreted that the program might not have been effective for children who deviated significantly from the target audience of this study. However, considering the larger number of positive impacts on children, the overall outcome favored benefits over drawbacks.

Teachers played a significant role in this program. Initially, they had to link each class's students’ accounts. Additionally, teachers who wished to do so could monitor and manage their students’ task performance. In practice, some teachers assisted students in task selection and even provided gift cards based on task completion rates.

Homeroom teachers gained a deeper understanding of the children through the ForME program. One teacher commented, “We couldn't know how our students spend their time at home from school. With the ForME program, we could learn how our class children spend their day.” We believe that gaining insight into a child's school and home life makes teaching more effective.

The program was notably beneficial for children facing difficulties in managing their daily lives. One comment highlighted, “Some parents care well, and their children grow up healthily, but there are also children on the opposite end. This program could be helpful for children facing difficulties.”

However, there were several opinions expressed such as, “The program's intention is good, but it is challenging for schoolteachers to manage children's lives at home,” and "With so many students to manage, I found it quite difficult to track individual student behavior using the prototype as the workload became difficult to manage." This situation calls for a thorough examination of how this program can operate effectively in educational settings.