MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling

Journaling—recording various personal matters ranging from observations and travels to overall daily experiences and thoughts [76]—takes place in various forms, including letter-style entries reminiscent of Anne Frank’s diary or more traditional prose. By narrating personal experiences and innermost feelings in a journal, people discover insights upon reflecting on their past events [89, 90], thereby improving mental wellness and quality of life [9, 70, 71, 91]. Researchers have found that journaling impacts mental health by fostering emotional inhibition [69], cognitive processing [36], and “freeing up” cognitive load [45, 74].

Beyond traditional pen and paper, research in HCI has explored how technology can augment a journaling method in articulating past emotions and experiences ( e.g. , [4, 10, 24, 37, 37, 42, 46]). Early studies in journaling focused on how technology can assist users in better documenting past events and experiences from life-logging perspectives [24, 37, 42]. For example, SenseCam has been proposed as a wearable ubiquitous computing device that utilizes camera sensor data to not only record but also reflect on the wearer’s daily life [37]. Subsequently, various attempts have been made to utilize contextual data and cues such as activity levels [46], mood [10], location, and photos [4] in journaling context.

Furthermore, HCI researchers have investigated ways to improve a journaling method to encourage users to express themselves in a manner that is more comfortable [34], engaging [58], and honest [68]. For example, Park et al. found that conversational agents create a social-like environment, encouraging self-reflection and enhancing expressive writing [68]. Furthermore, social support from online communities has been found to increase user participation in journaling activities [58]. Gonzales et al. [34] proposed an approach to mitigate the discomfort of revisiting negative memories using the sound generated from the data for more pleasurable [34].

Our work extends this line of research to enhance the journaling experience with technology. Specifically, we explore the potential of interactive dialogue with LLMs as a new format for journaling, allowing psychiatric patients to explore and reflect on their past experiences and emotions. To this end, we leverage a conversational agent as a complementary tool for assisting patients by providing prompt questions to engage users in deeper and more detailed documentation and reflection [46]. We particularly investigate how our approach plays a role in clinical settings as patient-generated health data, which we will cover in the following.

Patient-Generated Health Data (PGHD)—defined as “health-related data, such as health history, symptoms, biometric readings, treatment history, lifestyle choices, and other pertinent details, created, recorded, or gathered by patients” [81]—has increasingly become an essential tool in clinical settings to capture authentic, real-time insights into patients‘ health. Studies have shown that PGHD can enhance communication between patients and MHPs and offer contextual information about patients, thereby heightening MHPs’ awareness of patient health outside regular clinical visits [20, 55, 64]. For instance, photo journaling improved patient-provider communication for the management and treatment of irritable bowel syndrome by facilitating more effective discussions during treatment [18].

Within the mental health domain, PGHD range from structured mental health assessments (e.g., anxiety, depression) to more unstructured data, including mood-related symptoms and social interactions tracking (e.g., social media use, number of calls)[6, 21, 63]. Patient diaries or journals–“instructing the patient to write down one’s symptoms and other information related to one’s daily life to discuss them during clinical appointments” [27, 100]–can particularly be useful in mental health contexts as it can offer rich, self-documented insights, which could improve MHPs‘ understanding of their patients [95]. Despite the benefits of patient journaling, people often struggle with starting their entries, sticking to consistent journaling routines, and structuring their reflections [89]. Further, writing about emotions and past experiences can be intricate, as individuals vary in their capacity to recognize, interpret, and articulate their feelings [49, 57, 78]. For some, especially in psychotherapy, crafting a narrative that describes one’s life journey can be a challenging process [23, 72].

In this work, we aim to lower the barrier of journaling for patients by allowing them to carry on casual conversations with an AI instead of plain open-ended text writing. We demonstrate the potential of these dialogues as a source of PGHD in clinical settings to facilitate individual self-reflection and enhance communication between patients and MHPs.

The field of AI has proposed significant innovations in medical settings, such as aiding clinical decision-making and diagnosis [11, 60]. In a mental health domain, Natural Language Processing (NLP) techniques have been widely applied to venues that demand human language comprehension and generation for patient support and treatment [15, 22]. Since caring mental health often involves counseling, conversational agents, also known as chatbots, have particularly stood out in the mental health domain [1, 3]. Studies have demonstrated the potential of chatbots in facilitating different types of therapy, such as cognitive behavioral therapy [28, 29, 39], expressive writing [68], behavioral reinforcement [39], and solution-focused therapy [29]. Prior work has also shown that chatbots could ease the burden of disclosing sensitive information. Studies indicated that individuals may feel more comfortable communicating with chatbots because of the social stigma involved in communicating with human beings [52, 53, 68]. Furthermore, these approaches can help overcome temporal and spatial constraints, offering mental health support that is accessible at any time and anywhere [16].

Early mental health therapy chatbots predominantly relied upon rule-based or retrieval-based approaches [1, 2]. While these approaches provide a high level of control over the conversational flow, they fall short of carrying on open-ended conversations; that is, the chatbots neither respond to serendipitous topics nor offer versatile responses that are outside the scope of design [31, 38, 51]. In the mental health context, these limitations may impact the quality of the chatbot’s caring behaviors as their messages tend to be general [38, 68]. Recent LLMs have suggested a new paradigm of bootstrapping chatbots  [93, 94]. LLM-driven chatbots tend to produce human-like, context-aware responses for unseen topics [65, 94]. Consequently, LLM-driven chatbots excel in facilitating open-domain dialogues and engaging in unscripted interactions [8], offering flexibility and adaptability in conversations [73, 86], especially in complex scenarios like mental health support. For instance, GPT-3.5 demonstrates empathetic traits, such as recognizing emotions and offering emotionally supportive replies in various situations, predominantly in healthcare settings [84]. In certain cases, these models appear to have potential in tasks that involve empathy, showing promising results when compared to humans [7, 25]. Given these early findings, LLM-driven chatbots in the public health sector may offer some support in alleviating emotional burden and loneliness among isolated individuals, though this area is still under exploration [41]. Additionally, the integration of LLMs with human efforts in creating mental health peer support messages could potentially lead to more empathetic conversations [82].

However, LLMs suffer from inherent challenges tied to their transformer-based architecture [92]. One key issue is the explainability of the model output: it is challenging to discern how this ‘black box’ model interprets a given input prompt. As a result, designers struggle to predict both the LLM’s understanding of the input and the subsequent messages it might produce [54]. For instance, a chatbot leveraging GPT-2 for mental health therapy occasionally generated non-word sentences or produced more negative outputs than positive ones [93]. Replika, an LLM-driven application intended for mental well-being, has occasionally displayed harmful content and exhibited inconsistent conversational styles, undermining its role as a long-term companion [59].

In summary, our work leverages LLMs in two key components: First, we use an LLM to power a chatbot for patient journaling. Second, the clinician dashboard incorporates LLMs for various NLP tasks, such as text summarization and classification, to visualize summarized insights (e.g., [5, 32]) that are noteworthy for MHP’s treatment. Through this collaborative approach involving clinicians, our work explores a unique design space for designing deploying LLM-driven chatbots in the mental health domain, aiming to improve communication between patients and providers by facilitating the recording of daily experiences, which act as a bridge for a better understanding of patients.

We distributed recruitment flyers in the Department of Psychiatry at a local university hospital, inviting Mental Health Professionals (MHPs) working in departments of psychiatry and mental health care centers to participate. We recruited six MHPs (E1–6; two males and four females)—four clinical psychologists and two psychiatrists whose careers varied from 1 to 11 years. Four were clinical psychologists responsible for counseling and daily monitoring and intervention of at-risk patient groups in local mental health centers and university hospitals, and two were psychiatrists in charge of outpatient and inpatient ward treatment in the psychiatry department of university hospitals (see Table 1).

We invited participants to two 1-hour remote sessions on Zoom. Two researchers participated in the sessions. We first provided an overview of language model technologies and LLM’s natural language understanding and generation capabilities until we shared a common understanding of the principles, applications, opportunities, and limitations of LLMs. Considering that we were designing a system for individuals with mental health challenges, we thoroughly covered the drawbacks of LLMs, such as uncertainty in control and hallucinations.

After the overview, we went through group discussions on how LLMs could be utilized in the current patient treatment process. As a probe, we asked participants a focused set of questions on (1) the challenges MHPs currently face during patient treatment and counseling sessions, and (2) their expectations and envisioned opportunities of LLMs’ role in clinical mental health settings. We sought to understand the experts’ perspectives through questions such as, ‘What are the difficulties or challenges patients face in their daily lives between treatments (or counseling)?’, ‘What are the important considerations in self-care that patients perform in their daily lives?’, and ‘What questions or conversational techniques do you use to encourage patients to share about their daily lives and moods?’. The session was video recorded and later transcribed. We open-coded the transcripts to identify emerging themes. In the following, we cover the findings from the FGI.

Based on the lessons from the FGI, we refined the initial concept of MindfulDiary. We leveraged the conversational abilities of LLMs to help patients document their daily experiences between clinical visits. MHPs had access to the collected data to inform their clinical decision-making. Furthermore, both MHPs and the research team concur that LLMs should not act solely as the primary intervention due to their inherent limitations but should function as supportive tools for clinical consultations. The subsequent section outlines the design and development process of our system.

The MindfulDiary app for patients aims to support people who might have difficulty journaling due to apathy and cognitive load through naturalistic conversation driven by an LLM. The app consists of a home screen containing an introduction and guide to the system (Figure 2a), a journal writing screen (Figure 2b 2c), and a screen to review the diary entries (Figure 2d).

To ensure that MindfulDiary is reliable and safe for conversing with psychiatric patients, we underwent multiple rounds of pilot evaluation. First, we invited five psychiatrists and three clinical psychologists to test the conversational pipeline. The experts provided feedback on the instructions in the model prompts, focusing on their clinical relevance and the embedded terminology and strategies. Then, the experts inspected the chatbot’s behavior by chatting with it while role-playing as a patient persona. In particular, we examined the chatbot’s reactions to subtle implications of suicide or self-harm in user messages.

After iterating on the conversational pipeline, we conducted a pilot lab study with five patients admitted to a university hospital but about to be discharged soon. To ensure safety against risky messages from an LLM, we used a test platform where the participant’s clinician monitored the generated messages in real-time, approving them or sending better messages manually.

The clinician dashboard (c.f., Supplementary video) is a desktop application designed to facilitate monitoring patient’s journal entries and to provide analysis of the entries to help clinicians identify significant events, reactions, and emotions. The dashboard consists of the following components:

User Engagement. This section visualizes the participant’s overall engagement with MindfulDiary, including the number of journals written, the date and time they were written, and their length. The modified PHQ-9 scores for each session are also visualized, allowing professionals to track the user’s mental health trends using a validated tool.

Journals. This section displays the content of the journals written by patients. The information is presented in a card format, where each card offers a summary of the journal, including timestamps, total time taken to write the journal, and associated PHQ-9 score. The interaction logs between the patient and MindfulDiary are also provided in this section.

Insights. To assist professionals in browsing through the diary, this section visualizes (1) a word cloud to understand frequent terms that the participant used at a glance, (2) a summary of major events to highlight significant happenings and (3) summary of emotions to gauge the mood based on user input. When a specific period is selected for review, a comprehensive summary is generated. We used GPT-4 for most summarization tasks. To generate the word frequency data for the word cloud, we combined GPT-4 and a Korean morphological analysis package named Kiwi [50] to filter only nouns and verbs from the GPT output. Due to the limitations of language model-driven analysis, there might be occasional inaccuracies in the generated content. First-time users of this interface are alerted about possible inaccuracies. An in-interface tooltip also reminds users that the summarized outcomes might not be accurate.

MindfulDiary’s interface is developed using React, a JavaScript-based framework. The server, responsible for interfacing with the LLM and overseeing database operations, is implemented in Python. Google Firebase handles user authentication, data storage, and retrieval tasks. The conversational capabilities of MindfulDiary are powered by gpt-4, accessible through OpenAI’s API¹. We specifically used gpt-4-0613 model. For parameter setting, we consistently set the temperature to 0.7 and both a presence penalty and frequency penalty to 0.5.

We targeted outpatients from the Department of Mental Health at a University Hospital. Participants were selected based on specific criteria: (1) those who had been diagnosed with MDD and (2) those who did not exhibit heightened impulsive tendencies or harbor specific intentions towards self-harm or suicide. Key exclusion criteria included a history of psychotic disorders, substance-related disorders, neurodevelopmental disorders, and neurological disorders. Eligible participants were identified through evaluations conducted by psychiatrists. Flyers and consent forms were distributed to eligible patients. For minors, the consent form process was adhered to only when they were accompanied by a guardian at the hospital.

We compensated participants on a weekly basis of participation: For participating every seven days from the starting date, participants received 15,000 KRW (approx. 11 USD). If they completed the entire four-week study process, they received 20,000 KRW as a bonus (i.e., 80,000 KRW—approx. 60 USD—in total). We did not tie the number of dialogue entries to the compensation to ensure natural data entry behavior.

As a minimum requirement for study completion, we instructed the participants not to miss four consecutive days without conversing with MindfulDiary. If a participant missed three consecutive days, an experimenter sent a reminder. In cases where participants did not respond to these reminders, their participation in the study was discontinued. This procedure was implemented to ensure active monitoring and communication. Considering that our system is designed for individuals with mental health challenges, it was crucial to maintain contact with participants and ensure their adherence to the study protocol.

Initially, 36 patients started using MindfulDiary. During the deployment, eight dropped out as they did not meet the minimum data collection requirement. These participants were disengaged from MindfulDiary due to the lack of time or decreased interest. As a result, 28 participants (P1–28; 11 males and 17 females) completed the 4-week field study and were included in the analysis. The majority of participants were adolescents and adults, with ages ranging from 12 to 28 years, with an average age of 17.6 (SD = 3.26). Table 2 presents the demographic details and severity of depressive symptoms of the study participants. These scores are derived from psychiatric evaluations conducted within one week before the starting dates.

Figure 5 illustrates the procedure of the field deployment study. All interviews took place remotely on Zoom.

To explore participants’ usage patterns with MindfulDiary, we first conducted a descriptive statistics analysis. To determine any shifts in participants’ adherence over time, we examined weekly writing frequencies using a one-way repeated measures ANOVA (RM-ANOVA) with Greenhouse-Geisser correction. To gain a deeper qualitative understanding of the messages produced by MindfulDiary and interviews with patients and psychiatrists, we used open coding paired with thematic analysis [13]. For a more in-depth qualitative analysis of the messages produced by MindfulDiary and the interviews with patients and psychiatrists, we employed open coding paired with thematic analysis [13]. All interviews were audio-recorded and transcribed for this purpose.

The qualitative analysis was conducted by the first author, a PhD student in HCI, who open-coded the interview transcripts and interaction log data through multiple rounds of iteration. Another author who holds a PhD degree in HCI also contributed to this coding process. Following the initial coding, two psychiatrists reviewed the coded data to provide clinical insights and ensure the accuracy of interpretations. Through discussions among the research team, including these diverse perspectives, overarching themes were identified, enhancing the depth and validity of our qualitative findings.

Conducting this study, we are fully aware of the inherent risks associated with our research, particularly given the characteristics of participants diagnosed with MDD. To mitigate the risks, we first carefully screened participants, relying on evaluations conducted by psychiatrists. Individuals displaying heightened impulsive tendencies or harboring specific intentions towards self-harm or suicide were excluded from the study. In addition, participants were asked to take the PHQ-9 before interacting with MindfulDiary, along with an additional set of questions probing their recent attempts at self-harm or suicide. If a participant’s response to question number 9 of the PHQ-9, regarding suicidal/self-harm thoughts, scored ‘moderate or higher’ or if any recent suicide attempt was verified, the system pivoted to provide content geared towards alleviating anxiety and reducing stress rather than proceeding with the standard system. In such a case, a real-time alert was also sent to psychiatrists. Lastly, if sensitive themes frequently surfaced in a participant’s input during the study, their interactions with the system were temporarily halted. Psychiatrists subsequently re-evaluated such participants to assess the viability of their ongoing participation. During our experiment, for the case of P11, mentions of repetitive suicide and self-harm were detected. Consequently, an expert contacted the participant, the experiment was suspended for three days, and after a re-evaluation in an outpatient clinic, we resumed the system use with P11.

Further, to mitigate potential risks from the LLMs’ outputs, we embraced an iterative design methodology. The system’s interactions underwent repeated assessments to ensure it generated safe, non-harmful outputs. In addition, in the first week of each participant’s system use, all interactions between participants and MindfulDiary were observed in real time. To facilitate this process, when a participant started the session, the research team received a notification email. This notification included real-time monitoring links and reports of the survey responses that participants answered before each session. After the first week, user interactions and MindfulDiary were reviewed within a 12-hour window. During the review process, if an interaction contained sensitive content (specifically, terms pre-defined as sensitive by psychiatrists), the psychiatrists on our research team assessed the situation and contacted the affected participants if necessary.

Lastly, given that we were handling the patients’ personal and sensitive data, ensuring the secure protection and management of data was critical. Therefore, during the study, we utilized the Google Firebase authentication service to manage the user authentication process for participants. We were thus able to ensure that only authorized personnel had access to the data, and any attempts at unauthorized access could be promptly detected and managed. After the field study, all data was separated from personal identifiers to maintain anonymity.

Figure 6 summarizes the daily engagement of participants with MindfulDiary over the course of four weeks. The colored squares denote the days that participants conversed with MindfulDiary (i.e., days with interaction). Across four weeks, participants submitted 501 journal entries (17.90 entries per participant on average), 0.62 entries on average per day (more than once every two days). 22 out of 28 participants used MindfulDiary more than once every two days. Participants generally engaged with the app at a regular frequency, but we note that their engagement was also affected by the three-day-miss reminder and their visit to the clinic between Week 2 and 4. Each journaling session lasted an average of 438 seconds (around 7 minutes) but with notable individual variability (SD = 225.97). Each journal dialogue included messages with an average of 105.6 syllable count (SD = 49.41). Our analysis did not reveal significant differences in either the participants’ input length (F(1.735, 46.85) = 2.718, p = .084) or writing time (F(2.417, 65.25) = 2.549, p = .076) across the four different time points, as determined by the RM-ANOVA test. This suggests that users mostly retained a steady level of engagement during the four-week study.

Participants and MindfulDiary exchanged a total of 4,410 messages (i.e., 2,205 pairs of the AI and participant’s messages) during the field study. Each session consisted of 10.82 messages (SD = 2.70). Most exchanges between the AI and participants were carried on for an exploration of patients’ daily lives and emotions, as well as for casual conversations. In terms of the stage of the conversation, 62% (2,732 messages) of the messages were from Exploration, 30% (1220 messages) for Rapport building, and 6% (282 messages) for Sensitive topic. Only a small amount of messages were accounted for Wrapping up (62 messages) or not selected (14 messages).

To understand the contents that MindfulDiary generated, we delved deep into the content it generated. 72% of the AI messages took the form of questions, aiming to elicit responses about users’ daily experiences and emotions. We identified and categorized the primary strategies that MindfulDiary employed to assist patients’ journaling. There were four strategies employed by the LLM: Emotional Exploration, Activity/Behavior Exploration, In-depth Follow-up & Countermeasures, and Future Plan Exploration. For a comprehensive breakdown of these strategies, along with their descriptions and exemplar questions, refer to Table 3.

The average length of participants’ responses was 29.42 syllable counts, with a median of 20 (SD = 35.9). This suggests a left-skewed distribution, where many participants gave shorter responses and a smaller number provided considerably longer answers, causing a high variation. The minimum response length was one character, and the maximum was 559 syllable counts. We further conducted a qualitative analysis of these responses, seeking to identify the themes present in users’ interactions with the LLM. This allowed us to understand the scope and topics of the daily records that MindfulDiary collected from the patients.

Participants interacting with MindfulDiary conveyed a range of topics (see Table 4). They described a spectrum of emotional states, from negative feelings like exhaustion and anxiety to positive sentiments of pride and joy. Events and activities were recounted, offering insights into their daily routines, such as walking during school times or decreased activity post-vacation. They also shared thoughts and beliefs, sometimes related to current events, revealing patterns linked to mental health, like feelings of exclusion and loneliness. Regarding perceived health status, comments spanned from immediate ailments, such as headaches, to long-term health challenges. Distorted perceptions about their body included content on excessive dieting. Specifically, participants frequently discussed medications, revealing not just their physical reactions but also their perceptions and behaviors toward them. Some expressed concerns over the taste, while others mentioned adverse reactions from intake, like discomfort after swallowing multiple pills at once. Lastly, the realm of relationships & interactions had participants highlighting both the challenges and supports in their interpersonal connections, revealing their significant impact on mental well-being, from conflicts and trust issues to moments of affirmation and encouragement.

Overall, participants viewed MindfulDiary as a space where they could open up and share their stories, feeling a sense of empathy from the system. Participants particularly found the dialogue-driven interactions with MindfulDiary useful. One participant, P15, mentioned, “If it was just about recording daily activities or emotions like a regular diary, it might have been less engaging, and I could’ve found it tedious or might not have persisted for long. But this felt like having a conversation with AI, which added an element of fun and kept me engaged in continuous use.” Such a dialogue-driven journaling process aided participants in maintaining consistent records and helped in forming a habit consistent with our user engagement analysis. P7 stated, “I liked chatting with the AI at first, so I kept using it. The more I used it, the more it became a habit.”

In this section, we describe how MHPs utilized the clinician dashboard and the benefits and drawbacks of the system they reported, drawing on the debriefing interviews with the psychiatrists.

Our study highlighted the potential of MindfulDiary in clinical settings, mainly where adherence to interventions is important [62]. Core symptoms of depression, such as loss of energy, difficulty in carrying out mental processes, and feelings of apathy, often contribute to lower adherence to a professional’s advice or intervention [43]. Clinicians who participated in our FGI also highlighted these challenges in motivating patients to utilize the diary writing app. Our findings demonstrated that MindfulDiary helped mitigate these challenges by transforming the conventional journaling process into engaging conversations. Using MindfulDiary, users were able to engage in conversations with the system by answering prompts and questions, which made them feel the journaling process was more accessible and intriguing. This active participation ensures that the users are not overwhelmed by the task and are guided in documenting their feelings and experiences more richly.

Depression often locks patients into negative and rigid thought patterns [12]. Such patterns, resistance to change established thought paradigms, can severely limit a patient’s ability to perceive issues from multiple angles, leading to a harsh self-judgment [61]. Our study highlighted that the varied perspectives offered by LLM-driven chatbots like MindfulDiary could help challenge such fixed viewpoints [33]. By prompting users to revisit their initial evaluations or suggest alternative viewpoints, these chatbots could help break the cycle of cognitive rigidity. While our research underscores the promising role of LLM-driven chatbots in assisting psychiatric patients’ journaling process, it’s essential to note that these are preliminary findings. More work is needed to substantiate these findings in a clinical context.

Studies have suggested that sharing the data captured via chatbots with others, such as health professionals and family members, could further serve as an effective mediator that helps convey more truthful information [52, 56]. For instance, patients consistently displayed deep self-disclosure through chatbots, whether or not they intended to share their inputs with health professionals [52]. Aligned with prior work on PGHD [20, 64], MHPs in our study also perceived that MindfulDiary has shed light on patients’ daily events, emotions, and thoughts that might have been difficult to gain through regular clinical visits. This data offered MHPs valuable insights into the patient’s experiences and context.

Building on these findings, we could expand the potential presented by MindfulDiary in patient-provider communication. In the field of personal health informatics, existing research highlights the role of technology, such as photo journaling, in managing conditions like Irritable Bowel Syndrome. This tool not only empowers patients to record their daily experiences more effectively but also fosters enhanced collaboration between patients and healthcare providers [18, 79]. Such tools serve as vital artifacts in negotiating the boundaries of patient-provider interactions (i.e., boundary negotiating artifacts) [19].

This work adds a new dimension to this discussion by showing how LLM-assisted journaling lowers barriers to generating health data in daily life and fosters patient understanding. Specifically, we found that through this system, patients and providers can collaboratively reflect on mental health conditions. In the context of the stage-based model of personal informatics, the patient module in our MindfulDiary helps patients reduce the burden of collecting daily data and supports deeper recording. The expert module’s dashboard allows for the combined and transformed processes of diary data, survey data, and quantitative engagement data, supporting MHPs’ integration and reflection. Collaborative data generation and utilization with patients can enable care that reflects the patient’s values and the characteristics of their daily life. These insights serve as a basis for patient-provider collaboration.

However, our study findings underscore the importance of careful consideration in the clinical integration of systems like MindfulDiary. While we did not observe patients exaggerating their conditions or needs, this potential issue was raised as a concern by MHPs. They expressed apprehension about the possibility that sharing journal content with MHPs through MindfulDiary might lead some patients to exaggerate their conditions or needs. This concern highlights the need to consider not only the design of chatbots that facilitate patient disclosure behavior but also the complex dynamics between patients and providers in clinical settings. It is crucial to address these dynamics to ensure the effective and safe use of such technologies in mental health care. The growing prevalence of chatbots in mental health domains emphasizes the need for a holistic approach to their design and implementation. We highlight that engineers and MHPs need to collaborate closely, ensuring that these tools are not only technically sound but also tailored to meet the intricate dynamics of clinical settings [87].

In this section, we discuss the consideration for integrating LLMs into clinical mental health settings, drawing insights from the design and evaluation of MindfulDiary.

Aligning Domain Experts’ Expectations of LLMs. Developing and deploying MindfulDiary, we learned that aligning MHPs’ expectations with the capabilities and limitations of LLMs involves significant challenges. The capability of generative language models to improve mental health is difficult to measure in comparison with AI models in other medical domains, where objective metrics can determine performance. For instance, in medical imaging, AI can be evaluated based on its accuracy in identifying target diseases from MRI scans, using precise numerical percentages of correct identifications [83]. On the other hand, in the realm of mental health chatbots, gauging success is more nuanced, as it involves subjective interpretations of emotional well-being and psychological improvement, which cannot be easily quantified or compared in the same straightforward manner. This challenge is amplified in mental health, where soft skills like rapport building and emotional observation are important [30]. The use of LLMs in the mental health field is emerging, but little has been said about evaluating or defining the performance of models that are tailored to mental health. Our iterative evaluation process involving MHPs could inform researchers about how to develop and evaluate LLM-mediated mental health technology. When integrating into the clinical setting, this evaluation is also necessary for anticipating who the system would target and for what purpose it would be used. Hence, we advocate that engineers and researchers should carefully consider how to assist domain experts, who may lack AI expertise, in fully and accurately grasping the role and operation of LLM. It is also crucial for researchers and engineers to collaborate closely with these professionals to ensure the technology aligns with therapeutic needs and best practices [87].

Tailored LLM Evaluation for Clinical Mental Health Domains. The domain of mental health, which our study addresses, is characterized by the vulnerability of its target user group. The content discussed within this domain is often emotionally charged and sensitive. Therefore, prioritizing user safety becomes even more essential in this domain than in others. Considering the sensitivity of the domain, during our evaluation process, MHPs thoroughly tested the LLM’s output by trying out conversations on various sensitive topics in both implicit and explicit ways, drawing upon their clinical experiences. The contents the MHPs input were much more diverse and wide-ranging than what engineers could generate during the development. Additionally, MHPs showed concern that the hallucinations of the LLM could reinforce or expand the delusions of patients with delusional disorders. We highlight that developing evidence-based tests or benchmark sets to anticipate the behavior of the language models in collaboration with MHPs is critical when leveraging LLMs for clinical mental health settings.

Incorporating Perspectives of MHPs in Testing and Monitoring. Considering the caveats of current LLMs [47], it is critical to involve MHPs when deploying LLM-driven systems for patients in mental health contexts. While planning the field deployment study of MindfulDiary, we identified specific roles that MHPs could play. In the pre-use phase, MHPs should determine the suitability of users and facilitate the onboarding process with patients. During the mid-use phase, they should closely monitor interactions with the LLM and be prepared to intervene in cases of crises or unexpected use scenarios. Furthermore, they can offer or adjust treatments periodically based on long-term data. Additionally, they should regularly re-evaluate the continued use of the system. While some of these tasks should carefully be designed not to burden MHPs too much, it is important that LLMs do not make autonomous decisions about patients (e.g., diagnosis, prescription, or crisis management) but instead operate under professional oversight.

Providing Safeguards for Hallucinated LLM Generations . Our clinician dashboard provided various summarized information, such as word cloud, aggregating multiple dialogue entries so that the clinician quickly grasps the gist of the dialogues. Although we underwent intensive testing with the LLM-driven data summarizer, the LLM-driven data processing may still suffer from inaccuracies, biases, and misinterpretation [40, 75] of patient sentiments or context, which could adversely affect treatment decisions and patient well-being. To mitigate such drawbacks of LLMs in our study, we provided sufficient guidance to MHPs, cautioning them that the LLM-generated information they receive may be error-prone. However, in real-world settings, MHPs might accept the outputs of LLM without much attention. Therefore, when involving LLM-driven data processing, the system should foster careful reviewing of the content based on the expertise of MHPs. For example, future systems could incorporate features like highlighting in vivo phrases that were directly mentioned by patients and signify key aspects of their experience and feelings. By contrasting the in vivo phrases with the LLM’s original text, the system can encourage MHPs to put more scrutiny on the LLM’s original interpretation, which may contain errors, and the actual inputs spoken by patients.

Our recruitment method could impact the generalizability of our findings, as we recruited the patient participants for our field study from a single university hospital. Although we aimed to recruit patients with diverse types and levels of symptoms, our participants are not representative samples of psychiatric patients. They were young (mostly adolescents) and consulted by a fixed number of psychiatrists. While this work is just a first step toward designing an LLM-driven journaling app for psychiatric patients, further investigation is necessary with subjects from various backgrounds. To implement our pipeline, we used OpenAI’s GPT API, which provided the most capable LLM at the time of our study and was accessible via commercial API. As GPT models are continually updated, later models may not yield the same conversational behavior. To generalize the performance of our conversational pipeline design, future work is needed to compare multiple versions of MindfulDiary with different underlying LLMs.