research-article

Public Access

Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment

Authors:

Audrey Southerland,

Rebecca M. Jones,

James M. RehgAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 3

Article No.: 43, Pages 1 - 20

https://doi.org/10.1145/3131902

Published: 11 September 2017 Publication History

Abstract

Eye contact is a crucial element of non-verbal communication that signifies interest, attention, and participation in social interactions. As a result, measures of eye contact arise in a variety of applications such as the assessment of the social communication skills of children at risk for developmental disorders such as autism, or the analysis of turn-taking and social roles during group meetings. However, the automated measurement of visual attention during naturalistic social interactions is challenging due to the difficulty of estimating a subject’s looking direction from video. This paper proposes a novel approach to eye contact detection during adult-child social interactions in which the adult wears a point-of-view camera which captures an egocentric view of the child’s behavior. By analyzing the child’s face regions and inferring their head pose we can accurately identify the onset and duration of the child’s looks to their social partner’s eyes. We introduce the Pose-Implicit CNN, a novel deep learning architecture that predicts eye contact while implicitly estimating the head pose. We present a fully automated system for eye contact detection that solves the sub-problems of end-to-end feature learning and pose estimation using deep neural networks. To train our models, we use a dataset comprising 22 hours of 156 play session videos from over 100 children, half of whom are diagnosed with Autism Spectrum Disorder. We report an overall precision of 0.76, recall of 0.80, and an area under the precision-recall curve of 0.79, all of which are significant improvements over existing methods.

References

[1]

Michael Argyle and Janet Dean. 1965. Eye-contact, distance and affiliation. Sociometry (1965), 289--304.

[2]

Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2014. Continuous conditional neural fields for structured regression. In European Conference on Computer Vision. Springer, 593--608.

[3]

T Berry Brazelton, Edward Tronick, Lauren Adamson, Heidelise Als, and Susan Wise. 1975. Early mother-infant reciprocity. Parent-Infant Interaction 3 (1975), 137.

[4]

Leo Breiman. 2001. Random forests. Machine Learning 45, 1 (2001), 5--32.

Digital Library

[5]

Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 3 (2011), 27.

Digital Library

[6]

Katarzyna Chawarska and Frederick Shic. 2009. Looking but not seeing: Atypical visual scanning and recognition of faces in 2 and 4-year-old children with autism spectrum disorder. Journal of Autism and Developmental Disorders 39, 12 (2009), 1663.

[7]

Meia Chita-Tegmark. 2016. Social attention in ASD: a review and meta-analysis of eye-tracking studies. Research in Developmental Disabilities 48 (2016), 79--93.

[8]

Amy M Daniels and David S Mandell. 2014. Explaining differences in age at autism spectrum disorder diagnosis: A critical review. Autism 18, 5 (2014), 583--597.

[9]

Fernando De la Torre, Wen-Sheng Chu, Xuehan Xiong, Francisco Vicente, Xiaoyu Ding, and Jeffrey Cohn. 2015. Intraface. In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1--8.

[10]

Piotr Dollár, Peter Welinder, and Pietro Perona. 2010. Cascaded pose regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1078--1085.

[11]

Richard O Duda, Peter E Hart, and David G Stork. 2012. Pattern classification. John Wiley 8 Sons.

[12]

Sarah R Edmunds, Agata Rozga, Yin Li, Elizabeth A Karp, Lisa V Ibanez, James M Rehg, and Wendy L Stone. 2017. Brief Report: Using a Point-of-View Camera to Measure Eye Gaze in Young Children with Autism Spectrum Disorder During Naturalistic Social Interactions: A Pilot Study. Journal of Autism and Developmental Disorders (2017), 1--7.

[13]

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88, 2 (2010), 303--338.

Digital Library

[14]

Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. 2010. Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 9 (2010), 1627--1645.

Digital Library

[15]

Centers for Disease Control and Prevention. 2016. Summary of Autism Spectrum Disorder Prevalence Studies. https://www.cdc.gov/ncbddd/autism/documents/ASDPrevalenceDataTable2016.pdf. (2016). Accessed: 2017-05-03.

[16]

Tom Foulsham, Esther Walker, and Alan Kingstone. 2011. The where, what and when of gaze allocation in the lab and the natural environment. Vision Research 51, 17 (2011), 1920--1931.

[17]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.

Digital Library

[18]

Rebecca Grzadzinski, Themba Carr, Costanza Colombi, Kelly McGuire, Sarah Dufek, Andrew Pickles, and Catherine Lord. 2016. Measuring changes in social communication behaviors: preliminary development of the Brief Observation of Social Communication Change (BOSCC). Journal of Autism and Developmental Disorders 46, 7 (2016), 2464--2479.

[19]

R Grzadzinski, K Martinez, G Gunin, E Ajodan, S Kim, and C Lord. 2017. Development of the Brief Observation of Social Communication Change (BOSCC) for Verbally Able Children with ASD. Biennial Meeting of the Society for Research on Child Development (SRCD) (2017).

[20]

Quentin Guillon, Nouchine Hadjikhani, Sophie Baduel, and Bernadette Rogé. 2014. Visual social attention in autism spectrum disorder: Insights from eye tracking studies. Neuroscience 8 Biobehavioral Reviews 42 (2014), 279--297.

[21]

Dan Witzner Hansen and Qiang Ji. 2010. In the eye of the beholder: A survey of models for eyes and gaze. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 3 (2010), 478--500.

Digital Library

[22]

Mariko Hosozawa, Kyoko Tanaka, Toshiaki Shimizu, Tamami Nakano, and Shigeru Kitazawa. 2012. How children with specific language impairment view social situations: an eye tracking study. Pediatrics 129, 6 (2012), e1453--e1460.

[23]

Ted Hutman, Mandeep K Chela, Kristen Gillespie-Lynch, and Marian Sigman. 2012. Selective visual attention at twelve months: Signs of autism in early social interactions. Journal of Autism and Developmental Disorders 42, 4 (2012), 487--498.

[24]

Vidit Jain and Erik G Learned-Miller. 2010. Fddb: A benchmark for face detection in unconstrained settings. UMass Amherst Technical Report (2010).

[25]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia. ACM, 675--678.

Digital Library

[26]

Huaizu Jiang and Erik Learned-Miller. 2016. Face detection with the faster R-CNN. arXiv preprint arXiv:1606.03473 (2016).

[27]

Warren Jones, Katelin Carr, and Ami Klin. 2008. Absence of preferential looking to the eyes of approaching adults predicts level of social disability in 2-year-old toddlers with autism spectrum disorder. Archives of General Psychiatry 65, 8 (2008), 946--954.

[28]

Chris L Kleinke. 1986. Gaze and eye contact: a research review. Psychological Bulletin 100, 1 (1986), 78.

[29]

Ami Klin, Warren Jones, Robert Schultz, Fred Volkmar, and Donald Cohen. 2002. Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Archives of General Psychiatry 59, 9 (2002), 809--816.

[30]

Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. 2016. Eye tracking for everyone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2176--2184.

[31]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS). 1097--1105.

Digital Library

[32]

Michael Land and Benjamin Tatler. 2009. Looking and acting: vision and eye movements in natural behaviour. Oxford University Press.

[33]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Conference on Computer Vision. Springer, 740--755.

[34]

Catherine Lord, Pamela C DiLavore, and Katherine Gotham. 2012. Autism diagnostic observation schedule. Western Psychological Services Torrance, CA.

[35]

Silvia Magrelli, Patrick Jermann, Noris Basilio, François Ansermet, François Hentsch, Jaqueline Nadel, and Aude Billard. 2013. Social orienting of children with autism to facial expressions and speech: a study with a wearable eye-tracker in naturalistic settings. Frontiers in Psychology 4 (2013), 840.

[36]

P Mundy and C Françoise Acra. 2006. Joint attention, social engagement, and the development of social competence. The Development of Social Engagement: Neurobiological Perspectives (2006), 81--117.

[37]

Peter Mundy, Christine Delgado, Jessica Block, Meg Venezia, Anne Hogan, and Jeffrey Seibert. 2003. Early social communication scales (ESCS). Coral Gables, FL: University of Miami (2003).

[38]

Basilio Noris, Jacqueline Nadel, Mandy Barker, Nouchine Hadjikhani, and Aude Billard. 2012. Investigating gaze of children with ASD in naturalistic settings. PloS One 7, 9 (2012), e44144.

[39]

Opal Ousley, Rosa Arriaga, Michael Morrier, Jennifer Mathys, Monica Allen, and Gregory Abowd. 2013. Beyond parental report: findings from the rapid-abc, a new 4-minute interactive autism. Technical Report. Technical report, Georgia Institute of Technology.

[40]

Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep Face Recognition. In British Machine Vision Conference, Vol. 1. 6.

[41]

Karen Pierce, David Conant, Roxana Hazin, Richard Stoner, and Jamie Desmond. 2011. Preference for geometric patterns early in life as a risk factor for autism. Archives of General Psychiatry 68, 1 (2011), 101--109.

[42]

James Rehg, Gregory Abowd, Agata Rozga, Mario Romero, Mark Clements, Stan Sclaroff, Irfan Essa, O Ousley, Yin Li, Chanho Kim, and others. 2013. Decoding children’s social behavior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3414--3421.

Digital Library

[43]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS). 91--99.

Digital Library

[44]

Agata Rozga, Ted Hutman, Gregory S Young, Sally J Rogers, Sally Ozonoff, Mirella Dapretto, and Marian Sigman. 2011. Behavioral profiles of affected and unaffected siblings of children with autism: Contribution of measures of mother--infant interaction and nonverbal communication. Journal of Autism and Developmental Disorders 41, 3 (2011), 287--301.

[45]

Michael Rutter, A Le Couteur, and C Lord. 2003. Autism diagnostic interview-revised. Los Angeles, CA: Western Psychological Services 29 (2003), 30.

[46]

Noah J Sasson and Jed T Elison. 2012. Eye tracking young children with autism. Journal of Visualized Experiments 61 (2012), e3675--e3675.

[47]

Jeffrey S Shell, Roel Vertegaal, Daniel Cheng, Alexander W Skaburskis, Changuk Sohn, A James Stewart, Omar Aoudeh, and Connor Dickie. 2004. ECSGlasses and EyePliances: using attention to open sociable windows of interaction. In Proceedings of the 2004 Symposium on Eye Tracking Research 8 Applications (ETRA). ACM, 93--100.

Digital Library

[48]

Marian Sigman. 1998. The Emanuel Miller Memorial Lecture 1997: Change and continuity in the development of children with autism. Journal of Child Psychology and Psychiatry 39, 6 (1998), 817--827.

[49]

Marian Sigman, Peter Mundy, Tracy Sherman, and Judy Ungerer. 1986. Social interactions of autistic, mentally retarded and normal children and their caregivers. Journal of Child Psychology and Psychiatry 27, 5 (1986), 647--656.

[50]

Brian A Smith, Qi Yin, Steven K Feiner, and Shree K Nayar. 2013. Gaze locking: passive eye contact detection for human-object interaction. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 271--280.

Digital Library

[51]

Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato. 2014. Learning-by-synthesis for appearance-based 3d gaze estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1821--1828.

Digital Library

[52]

Matthew A Turk and Alex P Pentland. 1991. Face recognition using eigenfaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 586--591.

[53]

Paul Viola and Michael J Jones. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2 (2004), 137--154.

Digital Library

[54]

OMRON OKAO vision. 2017. https://www.omron.com/ecb/products/mobile/okao01.html. (2017). Accessed: 2017-05-03.

[55]

Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. 2015. From facial parts responses to face detection: A deep learning approach. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 3676--3684.

Digital Library

[56]

Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. 2016. WIDER FACE: A Face Detection Benchmark. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]

Zhefan Ye, Yin Li, Alireza Fathi, Yi Han, Agata Rozga, Gregory D Abowd, and James M Rehg. 2012. Detecting eye contact using wearable eye-tracking glasses. In Proceedings of the ACM Conference on Ubiquitous Computing. ACM, 699--704.

Digital Library

[58]

Zhefan Ye, Yin Li, Yun Liu, Chanel Bridges, Agata Rozga, and James M Rehg. 2015. Detecting bids for eye contact using a wearable camera. In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1--8.

[59]

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters 23, 10 (2016), 1499--1503.

[60]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. 2015. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4511--4520.

[61]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. 2016. It’s Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. arXiv preprint arXiv:1611.08860 (2016).

[62]

Lonnie Zwaigenbaum, Susan Bryson, and Nancy Garon. 2013. Early identification of autism spectrum disorders. Behavioral Brain Research 251 (2013), 133--146.

Cited By

Zarei MSolovov AGautam VFerreira BAssunção GMarin-Hernandez AMenezes P(2024)Transforming Collaboration: A Vision for Human-Aware Robots to Enhance Worker Safety and Boost Production Efficiency2024 IEEE 22nd Mediterranean Electrotechnical Conference (MELECON)10.1109/MELECON56669.2024.10608606(723-728)Online publication date: 25-Jun-2024
https://doi.org/10.1109/MELECON56669.2024.10608606
Karaca BSalah ADenissen JPoppe Rde Zwarte S(2024)Survey of Automated Methods for Nonverbal Behavior Analysis in Parent-Child Interactions2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10582009(1-11)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10582009
Boluk NKose H(2024)Evaluating Gaze Detection for Children with Autism Using the ChildPlay-R Dataset2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581976(1-5)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581976
Show More Cited By

Index Terms

Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment

Recommendations

Increasing the Social Interaction of Autism Child using Virtual Reality Intervention (VRI)
Nowadays there is an increase in number of autisms, a neuro-developmental disorder across the world. The level of autism varies with the symptoms such as inattention, interaction, social communication, repetitive behaviors, irritability and the like. ...
Parent and child problematic media use: The role of maternal postpartum depression and dysfunctional parent-child interactions in young children
Abstract
Problematic media use, or media use that interferes with daily functioning, is most often studied in adolescent or young adult age groups. Less research has examined problematic media use within the family system, among parents and ...
Highlights
- Problematic media use is seen in children as young as 3–4 years old and in their parents.
Detecting eye contact using wearable eye-tracking glasses
UbiComp '12: Proceedings of the 2012 ACM Conference on Ubiquitous Computing

We describe a system for detecting moments of eye contact between an adult and a child, based on a single pair of gaze-tracking glasses which are worn by the adult. Our method utilizes commercial gaze tracking technology to determine the adult's point ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 3

September 2017

2023 pages

EISSN:2474-9567

DOI:10.1145/3139486

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017

Accepted: 01 June 2017

Received: 01 May 2017

Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
1,447
Total Downloads

Downloads (Last 12 months)265
Downloads (Last 6 weeks)26

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zarei MSolovov AGautam VFerreira BAssunção GMarin-Hernandez AMenezes P(2024)Transforming Collaboration: A Vision for Human-Aware Robots to Enhance Worker Safety and Boost Production Efficiency2024 IEEE 22nd Mediterranean Electrotechnical Conference (MELECON)10.1109/MELECON56669.2024.10608606(723-728)Online publication date: 25-Jun-2024
https://doi.org/10.1109/MELECON56669.2024.10608606
Karaca BSalah ADenissen JPoppe Rde Zwarte S(2024)Survey of Automated Methods for Nonverbal Behavior Analysis in Parent-Child Interactions2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10582009(1-11)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10582009
Boluk NKose H(2024)Evaluating Gaze Detection for Children with Autism Using the ChildPlay-R Dataset2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581976(1-5)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581976
Uddin MShahriar MMahamood MAlnajjar FPramanik MAhad M(2024)Deep learning with image-based autism spectrum disorder analysis: A systematic reviewEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107185127(107185)Online publication date: Jan-2024
https://doi.org/10.1016/j.engappai.2023.107185
Pandya SJain SVerma J(2024)A comprehensive analysis towards exploring the promises of AI-related approaches in autism researchComputers in Biology and Medicine10.1016/j.compbiomed.2023.107801168(107801)Online publication date: Jan-2024
https://doi.org/10.1016/j.compbiomed.2023.107801
Kanwal AJaved KAli SRubab SKhan MAlasiry AMarzougui MShabaz M(2024)A hybrid framework for detection of autism using ConvNeXt-T and embedding clustersThe Journal of Supercomputing10.1007/s11227-023-05761-880:6(8156-8178)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s11227-023-05761-8
Wang JHu M(2023)The Application of Artificial Intelligence in the Recognition, Diagnosis, and Treatment of AutismProceedings of the 2023 4th International Symposium on Artificial Intelligence for Medicine Science10.1145/3644116.3644267(882-886)Online publication date: 20-Oct-2023
https://dl.acm.org/doi/10.1145/3644116.3644267
Tafasca SGupta AKojovic NGelsomini MMaillart TPapandrea MSchaer MOdobez J(2023)The AI4Autism Project: A Multimodal and Interdisciplinary Approach to Autism Diagnosis and StratificationCompanion Publication of the 25th International Conference on Multimodal Interaction10.1145/3610661.3616239(414-425)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3610661.3616239
Kalanadhabhatta MGanesan DRahman T(2023)Towards Accurate and Scalable Mental Health Screening Technologies for Young ChildrenAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610763(225-229)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610763
Song QDian RSun BXie JLi SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Multi-scale Conformer Fusion Network for Multi-participant Behavior AnalysisProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612847(9472-9476)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612847
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents