Work in Progress

Emotion Recognition via Face Tracking with RealSense(TM) 3D Camera for Children with Autism

Authors:

Tiffany Y. Tang,

Guanxing ChenAuthors Info & Claims

IDC '17: Proceedings of the 2017 Conference on Interaction Design and Children

Pages 533 - 539

https://doi.org/10.1145/3078072.3084321

Published: 27 June 2017 Publication History

Abstract

Although there is a growing recognition of the differences, not diminished abilities, of facial affective expressivity between Typically Developing (TD) and Autism Spectrum Disorder (ASD) individuals, which might lead to the varied recognizability of conveyed emotion by both TD and ASD individuals, little is explored on the ecological validity of these findings; that is, whether spontaneous affective facial expressions can better be produced and recognized by both populations. We aimed to address these issues in the present study, using children's cartoon clips to assess two aspects of spontaneous emotion production and recognition in a context closer to real-life children's cartoon movie watching (at home or a classroom). Based on the facial landmark data and a teacher/parent's manual emotion tags (happy), we performed a computational analysis to compare the happy emotion labels generated by the automated algorithm and the human TD rater. Two pilot studies of six ASD children revealed the potential as well as challenges of such an approach.

References

[1]

American Psychiatric Association. 2013. Diagnostic and statistical manual of mental disorders (5th ed.). Washington DC: American Psychiatric Association.

[2]

Simon Baron-Cohen. 200 Mind Reading: The Interactive Guide to Emotions. Jessica Kingsley Publisher, London, UK.

[3]

Sander Begeer, Hans M. Koot, Carolien Rieffe, Mark Meerum Terwogt, and Hedy Stegge. 2008. Emotional competence in children with autism: Diagnostic criteria and empirical evidence. Developmental Review 28,3: 342--369.

[4]

Rebecca Brewer, Federica Biotti, Carloline Catmur, Clare Press, Francesca Happé, Richard Cook, and Geoffrey Bird. 2016. Can Neurotypical Individuals Read Autistic Facial Expressions? Atypical Production of Emotional Facial Expressions in Autism Spectrum Disorders. Autism Research 9 2: 262--271.

[5]

Lisa Capps, Connie Kasari, Nurit Yirmiya, and Sigman, M. 1993. Parental perception of emotional expressiveness in children with autism. J. Consulting and Clinical Psychology 61, 3: 475--484.

[6]

Patrycja Czapinski, Susan E Bryson. 2003. Reduced facial muscle movements in autism: Evidence for dysfunction in the neuromuscular pathway? Brain and Cognition 51, 2: 177--179.

[7]

Daniel J. Faso, Noah J. Sasson, Amy E. Pinkham. 2015. Evaluating Posed and Evoked Facial Expressions of Emotion from Adults with Autism Spectrum Disorder. J. Autism and Developmental Disorders 45, 1: 75--89.

[8]

Ruth B. Grossman, Lisa R. Edelson, and Helen Tager-Flusberg. 2013. Emotional facial and vocal expressions during story retelling by children and adolescents with high-functioning autism. Journal of Speech, Language and Hearing Research 56, 3: 1035.

[9]

Ruth B Grossman. 2015. Judgments of social awkwardness from brief exposure to children with and without high-functioning autism. Autism 19, 5: 580--587.

[10]

Amanda C. Gulsrud, Laudan B. Jahromi, and Connie Kasari. 2010. The Co-regulation of emotions between mothers and their children with autism. J. Autism and Developmental Disorders 40, 2: 227--237.

[11]

Amy G. Halberstadt, Susanne A. Denham, and Julie C. Dunsmore. 2001. Affective Social Competence. Social Development 10, 1:79--119.

[12]

William J Hudenko, ad M. A. Magenheimer. 2011. Listeners prefer the laughs of children with autism to those of typically developing children. Autism 16, 6: 641--665.

[13]

Connie Kasari, Marian Sigman, Peter Mundy, and Nurit Yirmiya. 1990. Affective sharing in the context of joint attention interactions of normal, autistic, and mentally retarded children. J. Autism and Developmental Disorders, 20(1), 87--100.

[14]

Angeliki Metallinou, Ruth B. Grossman, and Shrikanth Narayanan. 2013. Quantifying atypicality in affective facial expressions of children with autism spectrum disorders. In Proc. of 2013 IEEE International Conference on Multimedia and Expo (ICME).

[15]

Rachel L. Moseley, Yury Shtyrov, Bettina Mohr, Michael V. Lombardo, Sim on Baron-Cohen, Friedemann Pulvermüller. 2015. Lost for emotion words: What motor and limbic brain activity reveals about autism and semantic theory. NeuroImage 104: 413--422.

[16]

Aparna Nadig, Holly Shaw. 2012. Acoustic and perceptual measurement of expressive prosody in high-functioning autism: Increased pitch range and what it means for listeners. Journal of Autism and Developmental Disorders 42, 499--511.

[17]

Rosalind Picard. 2009. Future affective technology for autism and emotion communication. Philosophical Transactions of the Royal Society B, 364:3575--3584.

[18]

Rosalind Picard. 2010. Emotion research by the people, for the people. Emotion Review 2, 3: 250--254.

[19]

James M. Rehg, Agata Rozga, Gregory D. Abowd and Matthew S. Goodwin. 2014. Behavioral Imaging and Autism, IEEE Pervasive Computing, 13, 2:84--87.

[20]

Vinícius Silva, Filomena Soares, João Sena Esteves, Joana Figueiredo, Cristina Santos and Ana Paula Pereira. 2017. Happiness and Sadness Recognition System-Preliminary Results with an Intel RealSense 3D Sensor. In Proc. of CONTROLO 2016, LNCS 402, Springer.

[21]

Margaret E. Snow, Margaret E. Hertzig and Theodore Shapiro. 1987. Expression of Emotion in Young Autistic Children. J. the American Academy of Child and Adolescent Psychiatry 26, 6: 836--838.

[22]

Steven D. Stagg, Rachel Slavny, Charlotte Hand, Alice Cardoso, and Pamela Smith. 2014. Does facial expressivity count? How typically developing children respond initially to children with autism. Autism 18, 6: 704--711.

[23]

Tiffany Y. Tang. 2016. Helping Neuro-typical individuals to read the emotion of children with autism spectrum disorder: an internet-of-things approach. In Proc. of the 15th ACM Interaction Design and Children Conference (ACM IDC2016), 666--671.

Digital Library

[24]

Martin A. Volker, Christopher Lopata, Donna A. Smith, Marcus L. Thomeer. 2009. Facial encoding of children with high-function autism spectrum disorders. Focus on Autism and Other Developmental Disabilities 24, 4: 195--204.

[25]

Amy K. Weimer, Amy M. Schatz, Alan Lincoln, Angela O Ballantyne, and Doris A. Trauner. 2001. "Motor" impairment in asperger syndrome: Evidence for a deficit in proprioception. Journal of Developmental & Behavioral Pediatrics 22,2: 92--101.

[26]

Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31, 1:39--58.

Digital Library

Cited By

Banos OComas-Gonzalez ZMedina JPolo AGil DPeral JAmador SVillalonga C(2024)Sensing technologies and machine learning methods for emotion recognition in autism: systematic reviewInternational Journal of Medical Informatics10.1016/j.ijmedinf.2024.105469(105469)Online publication date: May-2024
https://doi.org/10.1016/j.ijmedinf.2024.105469
Pepa LSpalazzi LCapecci MCeravolo M(2023)Automatic Emotion Recognition in Clinical Scenario: A Systematic Review of MethodsIEEE Transactions on Affective Computing10.1109/TAFFC.2021.312878714:2(1675-1695)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3128787
Briot KPizano ABouvard MAmestoy A(2021)New Technologies as Promising Tools for Assessing Facial Emotion Expressions Impairments in ASD: A Systematic ReviewFrontiers in Psychiatry10.3389/fpsyt.2021.63475612Online publication date: 5-May-2021
https://doi.org/10.3389/fpsyt.2021.634756
Show More Cited By

Index Terms

Emotion Recognition via Face Tracking with RealSense(TM) 3D Camera for Children with Autism
1. Human-centered computing

Recommendations

Facial expression recognition teaching to preschoolers with autism: a natural user interface approach
BCI '13: Proceedings of the 6th Balkan Conference in Informatics

The recognition of facial expressions is important for the perception of emotions. Understanding emotions is essential in human communication and social interaction. Children with autism have been reported to exhibit deficits in the recognition of ...
Validation of Emotions as a Measure of Selective Attention in Children with Autism Spectrum Disorder
ICEIT 2020: Proceedings of the 2020 9th International Conference on Educational and Information Technology

This research studied the validation of emotions as a measure of selective attention in children with autism spectrum disorder (ASD) and their typically developing (TD) peers. Seven basic emotions such as anger, sadness, disgust, joy, surprise, fear and ...
Position Paper: An Emoji-Based Interactive App to Aid Emotion Expression for Autistic Children
UbiComp '24: Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing

Autism spectrum disorder (ASD) is a neurodevelopmental condition typically characterized by two core characteristics as restricted, repetitive behaviors and impairment in social communication. As such, a large portion of the technology studies within the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IDC '17: Proceedings of the 2017 Conference on Interaction Design and Children

June 2017

808 pages

ISBN:9781450349215

DOI:10.1145/3078072

General Chairs:
Paulo Blikstein
Stanford University, USA
,
Dor Abrahamson
University of California-Berkeley, USA

Copyright © 2017 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2017

Check for updates

Author Tags

Qualifiers

Work in progress

Conference

IDC '17

Sponsor:

SIGCHI

IDC '17: Interaction Design and Children

June 27 - 30, 2017

California, Stanford, USA

Acceptance Rates

IDC '17 Paper Acceptance Rate 25 of 118 submissions, 21%;

Overall Acceptance Rate 172 of 578 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
371
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)1

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Banos OComas-Gonzalez ZMedina JPolo AGil DPeral JAmador SVillalonga C(2024)Sensing technologies and machine learning methods for emotion recognition in autism: systematic reviewInternational Journal of Medical Informatics10.1016/j.ijmedinf.2024.105469(105469)Online publication date: May-2024
https://doi.org/10.1016/j.ijmedinf.2024.105469
Pepa LSpalazzi LCapecci MCeravolo M(2023)Automatic Emotion Recognition in Clinical Scenario: A Systematic Review of MethodsIEEE Transactions on Affective Computing10.1109/TAFFC.2021.312878714:2(1675-1695)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3128787
Briot KPizano ABouvard MAmestoy A(2021)New Technologies as Promising Tools for Assessing Facial Emotion Expressions Impairments in ASD: A Systematic ReviewFrontiers in Psychiatry10.3389/fpsyt.2021.63475612Online publication date: 5-May-2021
https://doi.org/10.3389/fpsyt.2021.634756
Sharma KPapavlasopoulou SGiannakos MFails J(2019)Joint Emotional State of Children and Perceived Collaborative Experience in Coding ActivitiesProceedings of the 18th ACM International Conference on Interaction Design and Children10.1145/3311927.3323145(133-145)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1145/3311927.3323145
Sufi FGazzano JCalle FLopez J(2019)Multi-Camera Tracking System Applications Based on Reconfigurable Devices: A Review2019 International Conference on Computer, Communication, Chemical, Materials and Electronic Engineering (IC4ME2)10.1109/IC4ME247184.2019.9036575(1-5)Online publication date: Jul-2019
https://doi.org/10.1109/IC4ME247184.2019.9036575
Winoto PTang T(2019)Collaborative Online Educational Games for Children with Learning Disabilities: Lessons Learned2019 IEEE 5th International Conference on Collaboration and Internet Computing (CIC)10.1109/CIC48465.2019.00032(191-195)Online publication date: Dec-2019
https://doi.org/10.1109/CIC48465.2019.00032
Kalantarian HJedoui KWashington PTariq QDunlap KSchwartz JWall D(2019)Labeling Images with Facial EmotionArtificial Intelligence in Medicine10.1016/j.artmed.2019.06.004Online publication date: Jul-2019
https://doi.org/10.1016/j.artmed.2019.06.004
Winoto PTang TQiu XGuan A(2018)Assisting, Not Training, Autistic Children to Recognize and Share Each Other’s Emotions via Automatic Face-Tracking in a Collaborative Play EnvironmentUniversal Access in Human-Computer Interaction. Methods, Technologies, and Users10.1007/978-3-319-92049-8_46(628-636)Online publication date: 5-Jun-2018
https://doi.org/10.1007/978-3-319-92049-8_46
Tang TWinoto P(2018)An Internet of Things Approach to “Read” the Emotion of Children with Autism Spectrum DisorderInternet of Things A to Z10.1002/9781119456735.ch20(563-592)Online publication date: 31-May-2018
https://doi.org/10.1002/9781119456735.ch20

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents