short-paper

Public Access

Multimodal Dataset of Social Skills Training in Natural Conversational Setting

Authors:

Hidemi Iwasaka,

Yasuhiro Matsuda,

Tsubasa Morimoto,

Mitsuhiro Uratani,

Kosuke Okazaki,

Yuichiro Fujimoto,

Satoshi NakamuraAuthors Info & Claims

ICMI '21 Companion: Companion Publication of the 2021 International Conference on Multimodal Interaction

Pages 395 - 399

https://doi.org/10.1145/3461615.3485425

Published: 17 December 2021 Publication History

All formats PDF

Abstract

Social Skills Training (SST) is commonly used in psychiatric rehabilitation programs to improve social skills. It is especially effective for people who have social difficulties related to mental illnesses or developmental difficulties. Previous studies revealed several communication characteristics in Schizophrenia and Autism Spectrum Disorder. However, a few pieces of research have been conducted in natural conversational environments with computational features since automatic capture and analysis are difficult in natural settings. Even if the natural data collection is difficult, the data clearly have much better potential to identify the real communication characteristics of people with mental difficulties and the interaction differences between participants and trainers. Therefore, we collected a one-on-one SST multimodal dataset to investigate and automatically capture natural characteristics expressed by people who suffer from such mental difficulties as Schizophrenia or Autism Spectrum Disorder. To validate the potential of the dataset, using partially annotated data, we trained a classifier for Schizophrenia and healthy control with audio-visual features. We achieved over 85% accuracy, precision, recall, and f1-score in the classification task using only natural interaction data, instead of data captured in the specific tasks designed for clinical assessments.

References

[1]

Mohammad Rafayet Ali, Seyedeh Zahra Razavi, Raina Langevin, Abdullah Al Mamun, Benjamin Kane, Reza Rawassizadeh, Lenhart K. Schubert, and Mohammad Ehsan Hoque. 2020. A Virtual Conversational Agent for Teens with Autism Spectrum Disorder: Experimental Results and Design Lessons. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents(IVA ’20). Association for Computing Machinery, New York, NY, USA, Article 2, 8 pages. https://doi.org/10.1145/3383652.3423900

Digital Library

[2]

Mohammad Rafayet Ali, Kimberly Van Orden, Kimberly Parkhurst, Shuyang Liu, Viet-Duy Nguyen, Paul Duberstein, and M. Ehsan Hoque. 2018. Aging and Engaging: A Social Conversational Skills Training Program for Older Adults. In 23rd International Conference on Intelligent User Interfaces(IUI ’18). Association for Computing Machinery, New York, NY, USA, 55–66. https://doi.org/10.1145/3172944.3172958

Digital Library

[3]

T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L. Morency. 2018. OpenFace 2.0: Facial Behavior Analysis Toolkit. In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018). 59–66.

Digital Library

[4]

T. Baltrušaitis, M. Mahmoud, and P. Robinson. 2015. Cross-dataset learning and person-specific normalisation for automatic Action Unit detection. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). 1–6.

[5]

A. Bandura. 1969. Principles of behavior modification. Holt, Rinehart and Winston.

[6]

Alan S. Bellack, Kim T. Mueser, Susan Gingerich, and Julie Agresta. 2004. Social Skills Training for Schizophrenia: A Step-by-Step Guide (2 ed.). Guilford Press.

[7]

John N. Constantino and Christian P. Gruber. 2012. Social Responsiveness Scale, SRS-2(2 ed.). Western Psychological Services.

[8]

J. R. Dubno and M. F. Dorman. 1987. Effects of spectral flattening on vowel identification. J Acoust Soc Am 82, 5 (Nov 1987), 1503–1511.

[9]

O. Golan, S. Baron-Cohen, J. J. Hill, and Y. Golan. 2006. The ”reading the mind in films” task: complex emotion recognition in adults with and without autism spectrum conditions. Soc Neurosci 1, 2 (2006), 111–123.

[10]

I.L. Goldstein. 1986. Training in Organizations: Needs Assessment, Development and Evaluation. Brooks/Cole publishing company.

[11]

Mohammed Ehsan Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W. Picard. 2013. MACH: My Automated Conversation Coach. In Proceedings of UbiComp ’13. Association for Computing Machinery, New York, NY, USA, 697–706. https://doi.org/10.1145/2493432.2493502

Digital Library

[12]

S. R. Kay, A. Fiszbein, and L. A. Opler. 1987. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull 13, 2 (1987), 261–276.

[13]

R. S. Keefe, T. E. Goldberg, P. D. Harvey, J. M. Gold, M. P. Poe, and L. Coughenour. 2004. The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophr Res 68, 2-3 (Jun 2004), 283–297.

[14]

S. L. Kerr and J. M. Neale. 1993. Emotion perception in schizophrenia: specific deficit or further evidence of generalized poor performance?J Abnorm Psychol 102, 2 (May 1993), 312–318.

[15]

Akio Kikuchi. 1988. The development of a social skills scale. 38 (1988), 67–68. In Japanese.

[16]

Catherine Lord and Michael Rutter. 2012. Autism Diagnostic Observation Schedule, Second Edition. WPS.

[17]

A. Parola, A. Simonsen, V. Bliksted, and R. Fusaroli. 2020. Voice patterns in schizophrenia: A systematic review and Bayesian meta-analysis. Schizophr Res 216 (02 2020), 24–40.

[18]

T. L. Patterson, S. Moscona, C. L. McKibbin, K. Davidson, and D. V. Jeste. 2001. Social skills performance assessment among older patients with Schizophrenia. Schizophrenia Research 48, 2–3 (3 2001), 351–360.

[19]

Takeshi Saga, Hiroki Tanaka, Hidemi Iwasaka, and Satoshi Nakamura. 2020. Objective Prediction of Social Skills Level for Automated Social Skills Training Using Audio and Text Information. In Companion Publication of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, New York, NY, USA, 467–471. https://doi.org/10.1145/3395035.3425221

[20]

A. Salter. 1949. Conditioned reflex therapy. Creative Age Press.

[21]

S Sekimoto. 1982. Effects of formant peak emphasis on vowel intelligibility in frequency compressed. Annual Bulletin of logopedics and phoniatrics 16 (1982).

[22]

Theodore M. Singelis. 1994. The Measurement of Independent and Interdependent Self-Construals. Personality and Social Psychology Bulletin 20, 5 (1994), 580–591. https://doi.org/10.1177/0146167294205014

[23]

Quentin Summerfield, John Foster, Richard Tyler, and Peter J. Bailey. 1985. Influences of formant bandwidth and auditory frequency selectivity on identification of place of articulation in stop consonants. Speech Communication 4, 1 (1985), 213–229. https://doi.org/10.1016/0167-6393(85)90048-2

[24]

T. Sych, C. Casey, and P. Meadows. [n.d.]. Azure Kinect DK Documentation. https://docs.microsoft.com/en-us/azure/kinect-dk/ Last accessed: August 2021.

[25]

Hiroki Tanaka, Hidemi Iwasaka, Hideki Negoro, and Satoshi Nakamura. 2020. Analysis of conversational listening skills toward agent-based social skills training. Journal on Multimodal User Interfaces 14, 1 (01 Mar 2020), 73–82. https://doi.org/10.1007/s12193-019-00313-y

[26]

Hiroki Tanaka, Hideki Negoro, Hidemi Iwasaka, and Satoshi Nakamura. 2017. Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders. PLOS ONE 12, 8 (08 2017), 1–15. https://doi.org/10.1371/journal.pone.0182151

[27]

Vincent van Heuven. 2001. Praat, a system for doing phonetics by computer. Glot International 5, 9/10 (2001), 341–345.

[28]

Rohit Voleti, Stephanie Woolridge, Julie Liss, Melissa Milanovic, Christopher Bowie, and Visar Berisha. 2019. Objective Assessment of Social Skills Using Automated Language Analysis for Identification of Schizophrenia and Bipolar Disorder. In Proceedings of Interspeech 2019. International Speech Communication Association, 1433–1437. https://doi.org/10.21437/Interspeech.2019-2960

[29]

J. Wolpe. 1958. Psychotherapy by reciprocal inhibition. Stanford University Press.

Cited By

Tanaka HSaga TIwauchi KHonda MMorimoto TMatsuda YUratani MOkazaki KNakamura S(2023)The Validation of Automated Social Skills Training in Members of the General Population Over 4 Weeks: Comparative StudyJMIR Formative Research10.2196/448577(e44857)Online publication date: 27-Apr-2023
https://doi.org/10.2196/44857
Chuang CLin YLiu CLee LChang HLiu AHung SFu L(2023)Multimodal Assessment of Schizophrenia Symptom Severity From Linguistic, Acoustic and Visual CuesIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2023.330759731(3469-3479)Online publication date: 2023
https://doi.org/10.1109/TNSRE.2023.3307597
Saga TTanaka HMatsuda YMorimoto TUratani MOkazaki KFujimoto YNakamura S(2023)Automatic evaluation-feedback system for automated social skills trainingScientific Reports10.1038/s41598-023-33703-013:1Online publication date: 26-Apr-2023
https://doi.org/10.1038/s41598-023-33703-0
Show More Cited By

Index Terms

Multimodal Dataset of Social Skills Training in Natural Conversational Setting

Index terms have been assigned to the content through auto-classification.

Recommendations

A Virtual Conversational Agent for Teens with Autism Spectrum Disorder: Experimental Results and Design Lessons
IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

We present the design of an online social skills development interface for teenagers with autism spectrum disorder (ASD). The interface is intended to enable private conversation practice anywhere, anytime using a web-browser. Users converse informally ...
Virtual Social Robot Enhances the Social Skills of Children with HFA
Social Robotics
Abstract
Social skills are the skills that humans use to communicate with each other verbally and non-verbally. The deficit of social skills is a core symptom of children with autism spectrum disorder (ASD). Physical social robots and virtual environments ...
Applying the Game Mode and Teaching Strategies of Computational Thinking to the Improvement of Social Skills Training for Children with Autism Spectrum Disorders
Universal Access in Human-Computer Interaction. Applications and Practice
Abstract
The main congenital defects of children with autism spectrum disorders (ASD) are social communication barriers and behavioral defects in social reciprocity. Because it is difficult for them to distinguish the deeper social meanings or abstract ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '21 Companion: Companion Publication of the 2021 International Conference on Multimodal Interaction

October 2021

418 pages

ISBN:9781450384711

DOI:10.1145/3461615

Editors:
Zakia Hammal
Carnegie Mellon University
,
Carlos Busso
University of Texas at Dallas
,
Catherine Pelachaud
CNRS - ISIR, Sorbonne University
,
Sharon Oviatt
Monash University
,
Albert Ali Salah
Utrecht University and Boğaziçi University
,
Guoying Zhao
University of Oulu

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

Japan Science and Technology Agency

Conference

ICMI '21

Sponsor:

SIGCHI

ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 18 - 22, 2021

QC, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
230
Total Downloads

Downloads (Last 12 months)140
Downloads (Last 6 weeks)18

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tanaka HSaga TIwauchi KHonda MMorimoto TMatsuda YUratani MOkazaki KNakamura S(2023)The Validation of Automated Social Skills Training in Members of the General Population Over 4 Weeks: Comparative StudyJMIR Formative Research10.2196/448577(e44857)Online publication date: 27-Apr-2023
https://doi.org/10.2196/44857
Chuang CLin YLiu CLee LChang HLiu AHung SFu L(2023)Multimodal Assessment of Schizophrenia Symptom Severity From Linguistic, Acoustic and Visual CuesIEEE Transactions on Neural Systems and Rehabilitation Engineering10.1109/TNSRE.2023.330759731(3469-3479)Online publication date: 2023
https://doi.org/10.1109/TNSRE.2023.3307597
Saga TTanaka HMatsuda YMorimoto TUratani MOkazaki KFujimoto YNakamura S(2023)Automatic evaluation-feedback system for automated social skills trainingScientific Reports10.1038/s41598-023-33703-013:1Online publication date: 26-Apr-2023
https://doi.org/10.1038/s41598-023-33703-0
Saga TTanaka HMatuda YMorimoto TUratani MOkazaki KFujimoto YNakamura S(2022)Analysis of Feedback Contents and Estimation of Subjective Scores in Social Skills Training2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)10.1109/EMBC48229.2022.9871180(1086-1089)Online publication date: 11-Jul-2022
https://doi.org/10.1109/EMBC48229.2022.9871180

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents