research-article

Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery

Authors:

Andreas BullingAuthors Info & Claims

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

Pages 193 - 203

https://doi.org/10.1145/3126594.3126614

Published: 20 October 2017 Publication History

Abstract

Eye contact is an important non-verbal cue in social signal processing and promising as a measure of overt attention in human-object interactions and attentive user interfaces. However, robust detection of eye contact across different users, gaze targets, camera positions, and illumination conditions is notoriously challenging. We present a novel method for eye contact detection that combines a state-of-the-art appearance-based gaze estimator with a novel approach for unsupervised gaze target discovery, i.e. without the need for tedious and time-consuming manual data annotation. We evaluate our method in two real-world scenarios: detecting eye contact at the workplace, including on the main work display, from cameras mounted to target objects, as well as during everyday social interactions with the wearer of a head-mounted egocentric camera. We empirically evaluate the performance of our method in both scenarios and demonstrate its effectiveness for detecting eye contact independent of target object type and size, camera position, and user and recording environment.

Supplementary Material

suppl.mov (uistf2052-file3.mp4)

Supplemental video

Download
42.67 MB

MP4 File (p193-zhang.mp4)

Download
142.94 MB

References

[1]

Michael F Land and Mary Hayhoe. In what ways do eye movements contribute to everyday activities' Vision research, 41(25):3559--3565, 2001.

[2]

Chris L Kleinke. Gaze and eye contact: a research review. Psychological bulletin, 100(1):78, 1986.

[3]

Jeffrey S Shell, Ted Selker, and Roel Vertegaal. Interacting with groups of computers. Communications of the ACM, 46(3):40--46, 2003.

Digital Library

[4]

Zhefan Ye, Yin Li, Alireza Fathi, Yi Han, Agata Rozga, Gregory D Abowd, and James M Rehg. Detecting eye contact using wearable eye-tracking glasses. In Proceedings of the 2012 ACM conference on ubiquitous computing, pages 699--704. ACM, 2012.

Digital Library

[5]

Connor Dickie, Roel Vertegaal, David Fono, Changuk Sohn, Daniel Chen, Daniel Cheng, Jeffrey S Shell, and Omar Aoudeh. Augmenting and sharing memory with eyeblog. In Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences, pages 105--109. ACM, 2004.

Digital Library

[6]

Michita Imai, Tetsuo Ono, and Hiroshi Ishiguro. Physical relation and expression: Joint attention for human-robot interaction. IEEE Transactions on Industrial Electronics, 50(4):636--643, 2003.

[7]

Jeffrey S Shell, Roel Vertegaal, Daniel Cheng, Alexander W Skaburskis, Changuk Sohn, A James Stewart, Omar Aoudeh, and Connor Dickie. Ecsglasses and eyepliances: using attention to open sociable windows of interaction. In Proceedings of the 2004 symposium on Eye tracking research & applications, pages 93--100. ACM, 2004.

Digital Library

[8]

Yanxia Zhang, Ming Ki Chong, Jörg Müller, Andreas Bulling, and Hans Gellersen. Eye tracking for public displays in the wild. Personal and Ubiquitous Computing, 19(5--6):967--981, 2015.

[9]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. It's written all over your face: Full-face appearance-based gaze estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017.

[10]

Yusuke Sugano, Xucong Zhang, and Andreas Bulling. Aggregaze: Collective estimation of audience attention on public displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pages 821--831. ACM, 2016.

Digital Library

[11]

Jeffrey S Shell, Roel Vertegaal, and Alexander W Skaburskis. Eyepliances: attention-seeking devices that respond to visual attention. In CHI'03 extended abstracts on Human factors in computing systems, pages 770--771. ACM, 2003.

Digital Library

[12]

Roel Vertegaal, Connor Dickie, Changuk Sohn, and Myron Flickner. Designing attentive cell phone using wearable eyecontact sensors. In CHI'02 extended abstracts on Human factors in computing systems, pages 646--647. ACM, 2002.

Digital Library

[13]

Sarah R Edmunds, Agata Rozga, Yin Li, Elizabeth A Karp, Lisa V Ibanez, James M Rehg, and Wendy L Stone. Brief report: Using a point-of-view camera to measure eye gaze in young children with autism spectrum disorder during naturalistic social interactions: A pilot study. Journal of Autism and Developmental Disorders, pages 1--7, 2017.

[14]

Brian A Smith, Qi Yin, Steven K Feiner, and Shree K Nayar. Gaze locking: passive eye contact detection for human-object interaction. In Proceedings of the 26th annual ACM symposium on User interface software and technology, pages 271--280. ACM, 2013.

Digital Library

[15]

Zhefan Ye, Yin Li, Yun Liu, Chanel Bridges, Agata Rozga, and James M Rehg. Detecting bids for eye contact using a wearable camera. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on, volume 1, pages 1--8. IEEE, 2015.

[16]

Antje Nuthmann and John M Henderson. Object-based attentional selection in scene viewing. Journal of vision, 10(8):20--20, 2010.

[17]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4511--4520, 2015.

[18]

Paul P Maglio, Rob Barrett, Christopher S Campbell, and Ted Selker. Suitor: An attentive information system. In Proceedings of the 5th international conference on Intelligent user interfaces, pages 169--176. ACM, 2000.

Digital Library

[19]

Paul P Maglio, Teenie Matlock, Christopher S Campbell, Shumin Zhai, and Barton A Smith. Gaze and speech in attentive user interfaces. In Advances in Multimodal Interfaces ICMI 2000, pages 1--7. Springer, 2000.

Digital Library

[20]

Alice Oh, Harold Fox, Max Van Kleek, Aaron Adler, Krzysztof Gajos, Louis-Philippe Morency, and Trevor Darrell. Evaluating look-to-talk: a gaze-aware interface in a collaborative environment. In CHI'02 Extended Abstracts on Human Factors in Computing Systems, pages 650--651. ACM, 2002.

Digital Library

[21]

Eric Horvitz, Carl Kadie, Tim Paek, and David Hovel. Models of attention in computing and communication: from principles to applications. Communications of the ACM, 46(3):52--59, 2003.

Digital Library

[22]

Roel Vertegaal and Jeffrey S Shell. Attentive user interfaces: the surveillance and sousveillance of gaze-aware objects. Social Science Information, 47(3):275--298, 2008.

[23]

Frederik Brudy, David Ledo, Saul Greenberg, and Andreas Butz. Is anyone looking? mitigating shoulder surfing on public displays through awareness and protection. In Proceedings of The International Symposium on Pervasive Displays, page 1. ACM, 2014.

Digital Library

[24]

Kevin Smith, Sileye O Ba, Jean-Marc Odobez, and Daniel Gatica-Perez. Tracking the visual focus of attention for a varying number of wandering people. IEEE transactions on pattern analysis and machine intelligence, 30(7):1212--1229, 2008.

Digital Library

[25]

Dan Witzner Hansen and Qiang Ji. In the eye of the beholder: A survey of models for eyes and gaze. IEEE transactions on pattern analysis and machine intelligence, 32(3):478--500, 2010.

Digital Library

[26]

Carlos Hitoshi Morimoto, Arnon Amir, and Myron Flickner. Detecting eye position and gaze from a single camera and 2 light sources. In Pattern Recognition, 2002. Proceedings. 16th International Conference on, volume 4, pages 314--317. IEEE, 2002.

[27]

Zhiwei Zhu and Qiang Ji. Eye gaze tracking under natural head movements. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 918--923. IEEE, 2005.

Digital Library

[28]

Zhiwei Zhu, Qiang Ji, and Kristin P Bennett. Nonlinear eye gaze mapping function estimation via support vector regression. In Pattern Recognition, 2006. ICPR 2006. 18th International Conference on, volume 1, pages 1132--1135. IEEE, 2006.

Digital Library

[29]

Dan Witzner Hansen and Arthur EC Pece. Eye tracking in the wild. Computer Vision and Image Understanding, 98(1):155--181, 2005.

Digital Library

[30]

Feng Lu, Yusuke Sugano, Takahiro Okabe, and Yoichi Sato. Inferring human gaze from appearance via adaptive linear regression. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 153--160. IEEE, 2011.

Digital Library

[31]

Erroll Wood and Andreas Bulling. Eyetab: Model-based gaze estimation on unmodified tablet computers. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 207--210. ACM, 2014.

Digital Library

[32]

Jixu Chen, Qiang Ji, et al. A probabilistic approach to online eye gaze tracking without personal calibration. IEEE Transactions on Image Processing, 2014.

[33]

Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato. Appearance-based gaze estimation using visual saliency. IEEE transactions on pattern analysis and machine intelligence, 35(2):329--341, 2013.

Digital Library

[34]

Michael Xuelin Huang, Tiffany CK Kwok, Grace Ngai, Stephen CF Chan, and Hong Va Leong. Building a personalized, auto-calibrating eye tracker from user interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5169--5179. ACM, 2016.

Digital Library

[35]

Yusuke Sugano, Yasuyuki Matsushita, Yoichi Sato, and Hideki Koike. Appearance-based gaze estimation with online calibration from mouse operations. IEEE Transactions on Human-Machine Systems, 45(6):750--760, 2015.

[36]

Ben Benfold and Ian Reid. Unsupervised learning of a scene-specific coarse gaze estimator. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2344--2351. IEEE, 2011.

Digital Library

[37]

Isarun Chamveha, Yusuke Sugano, Daisuke Sugimura, Teera Siriteerakul, Takahiro Okabe, Yoichi Sato, and Akihiro Sugimoto. Head direction estimation from low resolution images with scene adaptation. Computer Vision and Image Understanding, 117(10):1502--1511, 2013.

Digital Library

[38]

Stefan Duffner and Christophe Garcia. Visual focus of attention estimation with unsupervised incremental learning. IEEE Transactions on Circuits and Systems for Video Technology, 26(12):2264--2272, 2016.

Digital Library

[39]

Ted Selker, Andrea Lockerd, and Jorge Martinez. Eye-r, a glasses-mounted eye motion detection interface. In CHI'01 extended abstracts on Human factors in computing systems, pages 179--180. ACM, 2001.

Digital Library

[40]

Connor Dickie, Roel Vertegaal, Jeffrey S Shell, Changuk Sohn, Daniel Cheng, and Omar Aoudeh. Eye contact sensing glasses for attention-sensitive wearable video blogging. In CHI'04 extended abstracts on Human factors in computing systems, pages 769--770. ACM, 2004.

Digital Library

[41]

John D Smith, Roel Vertegaal, and Changuk Sohn. Viewpointer: lightweight calibration-free eye tracking for ubiquitous handsfree deixis. In Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 53--61. ACM, 2005.

Digital Library

[42]

Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? In Advances in Neural Information Processing Systems, pages 199--207, 2015.

Digital Library

[43]

Adrià Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. Following gaze across views. arXiv preprint arXiv:1612.03094, 2016.

[44]

Kenneth Alberto Funes Mora, Florent Monay, and Jean-Marc Odobez. Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 255--258. ACM, 2014.

Digital Library

[45]

Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755--1758, 2009.

Digital Library

[46]

Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, pages 1--10. IEEE, 2016.

[47]

Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. Optics: ordering points to identify the clustering structure. In ACM Sigmod record, volume 28, pages 49--60. ACM, 1999.

Digital Library

[48]

Petros Xanthopoulos and Talayeh Razzaghi. A weighted support vector machine method for control chart pattern recognition. Computers & Industrial Engineering, 70:134--149, 2014.

Digital Library

[49]

Rainer Stiefelhagen, Jie Yang, and Alex Waibel. Modeling focus of attention for meeting indexing based on multiple cues. IEEE Transactions on Neural Networks, 13(4):928--938, 2002.

Digital Library

[50]

Michael Voit and Rainer Stiefelhagen. Deducing the visual focus of attention from head pose estimation in dynamic multi-view meeting scenarios. In Proceedings of the 10th international conference on Multimodal interfaces, pages 173--180. ACM, 2008.

Digital Library

[51]

ByungIn Yoo, Jae-Joon Han, Changkyu Choi, Kwonju Yi, Sungjoo Suh, Dusik Park, and Changyeong Kim. 3d user interface combining gaze and hand gestures for large-scale display. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, pages 3709--3714. ACM, 2010.

Digital Library

Cited By

Wu YWang YQu SLi ZChen GLarson K(2024)ELF-UAProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/161(1452-1461)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/161
Ma FHe YSun BLi SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Less is More: Adaptive Feature Selection and Fusion for Eye Contact DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3688987(11390-11396)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3688987
Elfares MReisert PHu ZTang WKüsters RBulling A(2024)PrivatEyes: Appearance-based Gaze Estimation Using Federated Secure Multi-Party ComputationProceedings of the ACM on Human-Computer Interaction10.1145/36556068:ETRA(1-23)Online publication date: 28-May-2024
https://dl.acm.org/doi/10.1145/3655606
Show More Cited By

Index Terms

Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery
1. Human-centered computing

Recommendations

Robust eye contact detection in natural multi-person interactions using gaze and speaking behaviour
ETRA '18: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications

Eye contact is one of the most important non-verbal social cues and fundamental to human interactions. However, detecting eye contact without specialised eye tracking equipment poses significant challenges, particularly for multiple people in real-world ...
GAZE-2: conveying eye contact in group video conferencing using eye-controlled camera direction
CHI '03: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

GAZE-2 is a novel group video conferencing system that uses eye-controlled camera direction to ensure parallax-free transmission of eye contact. To convey eye contact, GAZE-2 employs a video tunnel that allows placement of cameras behind participant ...
Detecting eye contact using wearable eye-tracking glasses
UbiComp '12: Proceedings of the 2012 ACM Conference on Ubiquitous Computing

We describe a system for detecting moments of eye contact between an adult and a child, based on a single pair of gaze-tracking glasses which are worn by the adult. Our method utilizes commercial gaze tracking technology to determine the adult's point ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

October 2017

870 pages

ISBN:9781450349819

DOI:10.1145/3126594

General Chair:
Krzysztof Gajos
Harvard University
,
Program Chairs:
Jennifer Mankoff
Carnegie Mellon University
,
Chris Harrison
Carnegie Mellon University

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

JST CREST
EXC-MMCI at Saarland University

Conference

UIST '17

Sponsor:

UIST '17: The 30th Annual ACM Symposium on User Interface Software and Technology

October 22 - 25, 2017

QC, Québec City, Canada

Acceptance Rates

UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
913
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)8

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu YWang YQu SLi ZChen GLarson K(2024)ELF-UAProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/161(1452-1461)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/161
Ma FHe YSun BLi SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Less is More: Adaptive Feature Selection and Fusion for Eye Contact DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3688987(11390-11396)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3688987
Elfares MReisert PHu ZTang WKüsters RBulling A(2024)PrivatEyes: Appearance-based Gaze Estimation Using Federated Secure Multi-Party ComputationProceedings of the ACM on Human-Computer Interaction10.1145/36556068:ETRA(1-23)Online publication date: 28-May-2024
https://dl.acm.org/doi/10.1145/3655606
Dembinsky DWatanabe KDengel AIshimaru S(2024)Eye Movement in a Controlled Dialogue SettingProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3653337(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3653337
Wu ZWang CSun GFu Z(2024)A Gaze Estimation Method Based on Binocular CamerasInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142335001338:01Online publication date: 1-Feb-2024
https://doi.org/10.1142/S0218001423350013
Cheng YWang HBao YLu F(2024)Appearance-Based Gaze Estimation With Deep Learning: A Review and BenchmarkIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.339357146:12(7509-7528)Online publication date: Dec-2024
https://doi.org/10.1109/TPAMI.2024.3393571
Ghosh SDhall AHayat MKnibbe JJi Q(2024)Automatic Gaze Analysis: A Survey of Deep Learning Based ApproachesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.332133746:1(61-84)Online publication date: Jan-2024
https://doi.org/10.1109/TPAMI.2023.3321337
Wu YCHen GYe LJia YLiu ZWang Y(2024)TTAGaze: Self-Supervised Test-Time Adaptation for Personalized Gaze EstimationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341224334:11(10959-10971)Online publication date: Nov-2024
https://doi.org/10.1109/TCSVT.2024.3412243
Hu DHuang K(2024)Semi-Supervised Multitask Learning Using Gaze Focus for Gaze EstimationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.338359734:9(7935-7946)Online publication date: Sep-2024
https://doi.org/10.1109/TCSVT.2024.3383597
Luo HGao PLi JZeng D(2024)Gaze Estimation Based on the Improved Xception NetworkIEEE Sensors Journal10.1109/JSEN.2024.335908524:6(8450-8464)Online publication date: 15-Mar-2024
https://doi.org/10.1109/JSEN.2024.3359085
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents