research-article

Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics

Authors:

Guillermo Cecchi, and

Djallel BouneffoufAuthors Info & Claims

WWW '23 Companion: Companion Proceedings of the ACM Web Conference 2023

April 2023

Pages 932 - 939

https://doi.org/10.1145/3543873.3587623

Published: 30 April 2023 Publication History

Abstract

We introduce a Reinforcement Learning Psychotherapy AI Companion that generates topic recommendations for therapists based on patient responses. The system uses Deep Reinforcement Learning (DRL) to generate multi-objective policies for four different psychiatric conditions: anxiety, depression, schizophrenia, and suicidal cases. We present our experimental results on the accuracy of recommended topics using three different scales of working alliance ratings: task, bond, and goal. We show that the system is able to capture the real data (historical topics discussed by the therapists) relatively well, and that the best performing models vary by disorder and rating scale. To gain interpretable insights into the learned policies, we visualize policy trajectories in a 2D principal component analysis space and transition matrices. These visualizations reveal distinct patterns in the policies trained with different reward signals and trained on different clinical diagnoses. Our system’s success in generating DIsorder-Specific Multi-Objective Policies (DISMOP) and interpretable policy dynamics demonstrates the potential of DRL in providing personalized and efficient therapeutic recommendations.

References

[1]

Julius Adorf. 2013. Web speech API. KTH Royal Institute of Technology (2013), 1–11.

[2]

Charu C Aggarwal 2016. Recommender systems. Vol. 1. Springer.

[3]

Chumki Basu, Haym Hirsh, William Cohen, 1998. Recommendation as classification: Using social and content-based information in recommendation. In Aaai/iaai. 714–720.

[4]

Edward S Bordin. 1979. The generalizability of the psychoanalytic concept of the working alliance.Psychotherapy: Theory, research & practice3 (1979), 252.

[5]

Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics 8 (2020), 439–453.

[6]

Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596.

[7]

Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without exploration. In International conference on machine learning. PMLR, 2052–2062.

[8]

Sarah Graham, Colin Depp, Ellen E Lee, Camille Nebeker, Xin Tu, Ho-Cheol Kim, and Dilip V Jeste. 2019. Artificial intelligence for mental health and mental illnesses: an overview. Current psychiatry reports 21, 11 (2019), 1–18.

[9]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. 173–182.

Digital Library

[10]

Adam O Horvath. 1981. An exploratory study of the working alliance: Its measurement and relationship to therapy outcome. (1981).

[11]

Yehuda Koren, Steffen Rendle, and Robert Bell. 2022. Advances in collaborative filtering. Recommender systems handbook (2022), 91–142.

[12]

Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning. PMLR, 1188–1196.

[13]

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020).

[14]

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1419–1428.

Digital Library

[15]

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).

[16]

Baihan Lin. 2022. Computational Inference in Cognitive Science: Operational, Societal and Ethical Considerations. arXiv preprint arXiv:2210.13526 (2022).

[17]

Baihan Lin. 2022. Knowledge Management System with NLP-Assisted Annotations: A Brief Survey and Outlook. In CIKM Workshops.

[18]

Baihan Lin. 2022. Personality Effect on Psychotherapy Outcome: A Predictive Natural Language Processing Framework. arXiv preprint (2022).

[19]

Baihan Lin. 2022. Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook. arXiv preprint arXiv:2210.13623 (2022).

[20]

Baihan Lin. 2022. Voice2Alliance: automatic speaker diarization and quality assurance of conversational alignment. In INTERSPEECH.

[21]

Baihan Lin, Djallel Bouneffouf, and Guillermo Cecchi. 2022. Predicting human decision making in psychological tasks with recurrent neural networks. PloS one (2022).

[22]

Baihan Lin, Djallel Bouneffouf, and Guillermo Cecchi. 2022. Predicting Human Decision Making with LSTM. In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE.

[23]

Baihan Lin, Djallel Bouneffouf, Guillermo Cecchi, and Ravi Tejwani. 2023. Neural topic modeling of psychotherapy sessions. In International Workshop on Health Intelligence. Springer.

[24]

Baihan Lin, Guillermo Cecchi, and Djallel Bouneffouf. 2022. SupervisorBot: NLP-Annotated Real-Time Recommendations of Psychotherapy Treatment Strategies with Deep Reinforcement Learning. arXiv preprint arXiv:2208.13077 (2022).

[25]

Baihan Lin, Guillermo Cecchi, and Djallel Bouneffouf. 2022. Working Alliance Transformer for Psychotherapy Dialogue Classification. arXiv preprint arXiv:2210.15603 (2022).

[26]

Baihan Lin, Guillermo Cecchi, and Djallel Bouneffouf. 2023. Deep Annotation of Therapeutic Working Alliance in Psychotherapy. In International Workshop on Health Intelligence. Springer.

[27]

Baihan Lin, Guillermo Cecchi, and Djallel Bouneffouf. 2023. Helping Therapists with NLP-Annotated Recommendation. In Joint Proceedings of the ACM IUI Workshops.

[28]

Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. 2020. A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 744–752.

[29]

Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. 2020. Unified models of human behavioral agents in bandits, contextual bandits and rl. arXiv preprint arXiv:2005.04544 (2020).

[30]

Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. 2021. Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL. In International Workshop on Human Brain and Artificial Intelligence. Springer, 14–33.

[31]

Baihan Lin, Stefan Zecevic, Djallel Bouneffouf, and Guillermo Cecchi. 2023. TherapyView: Visualizing Therapy Sessions with Temporal Topic Modeling and AI-Generated Arts. arXiv preprint arXiv:2302.10845 (2023).

[32]

Baihan Lin and Xinxin Zhang. 2020. Speaker diarization as a fully online learning problem in MiniVox. arXiv preprint arXiv:2006.04376 (2020).

[33]

Baihan Lin and Xinxin Zhang. 2020. VoiceID on the fly: A speaker recognition system that learns from scratch. In INTERSPEECH.

[34]

Baihan Lin and Xinxin Zhang. 2021. Speaker Diarization as a Fully Online Bandit Learning Problem in MiniVox. In ACML. 1660–1674.

[35]

Baihan Lin and Xinxin Zhang. 2022. iSpeak: Interactive Spoken Language Understanding System For Children with Speech and Language Disorders. In 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE.

[36]

Baihan Lin and Xinxin Zhang. 2023. A Reinforcement Learning Framework for Online Speaker Diarization. arXiv preprint arXiv:2302.10924 (2023).

[37]

Malte Ludewig and Dietmar Jannach. 2018. Evaluation of session-based recommendation algorithms. User Modeling and User-Adapted Interaction 28, 4 (2018), 331–390.

Digital Library

[38]

Daniel J Martin, John P Garske, and M Katherine Davis. 2000. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review.Journal of consulting and clinical psychology 68, 3 (2000), 438.

[39]

Tara Matthews, Kathleen O’Leary, Anna Turner, Manya Sleeper, Jill Palzkill Woelfer, Martin Shelton, Cori Manthorne, Elizabeth F Churchill, and Sunny Consolvo. 2017. Stories from survivors: Privacy & security practices when coping with intimate partner abuse. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2189–2201.

Digital Library

[40]

Vikram Patel, Shekhar Saxena, Crick Lund, Graham Thornicroft, Florence Baingana, Paul Bolton, Dan Chisholm, Pamela Y Collins, Janice L Cooper, Julian Eaton, 2018. The Lancet Commission on global mental health and sustainable development. The lancet 392, 10157 (2018), 1553–1598.

[41]

Michael J Pazzani and Daniel Billsus. 2007. Content-based recommendation systems. In The adaptive web. Springer, 325–341.

Digital Library

[42]

Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of international conference on World Wide Web. 285–295.

Digital Library

[43]

Anand Satiani, Julie Niedermier, Bhagwan Satiani, and Dale P Svendsen. 2018. Projected workforce of psychiatrists in the United States: a population analysis. Psychiatric Services 69, 6 (2018), 710–713.

[44]

Xiaoyuan Su and Taghi M Khoshgoftaar. 2009. A survey of collaborative filtering techniques. Advances in artificial intelligence 2009 (2009).

[45]

Terence J Tracey and Anna M Kokotovic. 1989. Factor structure of the working alliance inventory.Psychological Assessment: A journal of consulting and clinical psychology 1, 3 (1989), 207.

[46]

Bruce E Wampold. 2015. How important are the common factors in psychotherapy? An update. World Psych 14 (2015), 270–277.

[47]

Xiaomei Wang, Sudeep Hegde, Changwon Son, Bruce Keller, Alec Smith, Farzan Sasangohar, 2020. Investigating mental health of US college students during the COVID-19 pandemic: cross-sectional survey study. Journal of medical Internet research 22, 9 (2020), e22817.

[48]

Xinxi Wang, Yi Wang, David Hsu, and Ye Wang. 2014. Exploration in interactive personalized music recommendation: a reinforcement learning approach. ACM TOMM 11, 1 (2014), 1–22.

[49]

C Edward Watkins Jr. 2013. Being and becoming a psychotherapy supervisor: The crucial triad of learning difficulties. American Journal of Psychotherapy 67, 2 (2013), 134–150.

[50]

Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-based recommendation with graph neural networks. In Proceedings of AAAI, Vol. 33. 346–353.

Digital Library

[51]

Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A deep reinforcement learning framework for news recommendation. In Proceedings of the 2018 world wide web conference. 167–176.

Digital Library

[52]

Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. 2020. Pseudo Dyna-Q: A reinforcement learning framework for interactive recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining. 816–824.

Digital Library

Cited By

Šafran VLin SNateqi JMartin ASmrke UAriöz UPlohl NRojc MBēma DChávez MHorvat MMlakar I(2024)Multilingual Framework for Risk Assessment and Symptom Tracking (MRAST)Sensors10.3390/s2404110124:4(1101)Online publication date: 8-Feb-2024
https://doi.org/10.3390/s24041101
Lin B(2024)Reinforcement learning and bandits for speech and language processingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122254238:PEOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122254
Lin BCecchi GBouneffouf DElkind E(2023)SupervisorBotProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/837(7149-7153)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/837
Show More Cited By

Index Terms

Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics

Recommendations

Deep reinforcement learning for page-wise recommendations
RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

Recommender systems can mitigate the information overload problem by suggesting users' personalized items. In real-world recommendations such as e-commerce, a typical interaction between the system and its users is - users are recommended a page of ...
Read More
Attitudes and perspectives towards the preferences for artificial intelligence in psychotherapy
Abstract
The use of artificial intelligence (AI) in psychotherapy has been increased in recent years. While these technologies in psychotherapy are growing, the circumstances of accepting artificial tools during psychotherapy need to be ...
Highlights
- We explored the factors of choosing AI-based psychotherapy.
- The less stigma and ...
Read More
Multi-layer Attention Social Recommendation System Based on Deep Reinforcement Learning
Knowledge Science, Engineering and Management
Abstract
The recommendation system based on deep reinforcement learning recommends interesting content to users through the interaction of recommendation agents and usersThis is to inform you that corresponding author has been identified as per the ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '23 Companion: Companion Proceedings of the ACM Web Conference 2023

April 2023

1567 pages

ISBN:9781450394192

DOI:10.1145/3543873

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '23

Sponsor:

SIGWEB

WWW '23: The ACM Web Conference 2023

April 30 - May 4, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
115
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)16

Other Metrics

View Author Metrics

Citations

Cited By

Šafran VLin SNateqi JMartin ASmrke UAriöz UPlohl NRojc MBēma DChávez MHorvat MMlakar I(2024)Multilingual Framework for Risk Assessment and Symptom Tracking (MRAST)Sensors10.3390/s2404110124:4(1101)Online publication date: 8-Feb-2024
https://doi.org/10.3390/s24041101
Lin B(2024)Reinforcement learning and bandits for speech and language processingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122254238:PEOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122254
Lin BCecchi GBouneffouf DElkind E(2023)SupervisorBotProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/837(7149-7153)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/837
Manickam NPonnusamy V(2023)A Comprehensive Analysis of Psychiatric Disorders Using Deep Learning2023 3rd International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON)10.1109/SMARTGENCON60755.2023.10442235(1-8)Online publication date: 29-Dec-2023
https://doi.org/10.1109/SMARTGENCON60755.2023.10442235

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents