Article

Estimating focus of attention based on gaze and sound

Authors:

Rainer Stiefelhagen,

Alex WaibelAuthors Info & Claims

PUI '01: Proceedings of the 2001 workshop on Perceptive user interfaces

Pages 1 - 9

https://doi.org/10.1145/971478.971505

Published: 15 November 2001 Publication History

Abstract

Estimating a person's focus of attention is useful for various human-computer interaction applications, such as smart meeting rooms, where a user's goals and intent have to be monitored. In work presented here, we are interested in modeling focus of attention in a meeting situation. We have developed a system capable of estimating participants' focus of attention from multiple cues. We employ an omnidirectional camera to simultaneously track participants' faces around a meeting table and use neural networks to estimate their head poses. In addition, we use microphones to detect who is speaking. The system predicts participants' focus of attention from acoustic and visual information separately, and then combines the output of the audio- and video-based focus of attention predictors. We have evaluated the system using the data from three recorded meetings. The acoustic information has provided 8% error reduction on average compared to using a single modality.

References

[1]

G. D. Abowd, C. Atkeson, A. Feinstein, C. Hmelo, R. Kooper, S. Long, N. Sawhney, and M. Tani. Teaching and learning as multimedia authoring: The classroom 2000 project. In Proceedings of the ACM Multimedia'96 Conference, pages 187--198, November 1996.

Digital Library

[2]

M. Argyle and M. Cook. Gaze and Mutual Gaze. Cambridge University Press, 1976.

[3]

C. M. Bishop. Neural Networks for Pattern Recognition. Clarendon Press, 1995.

Digital Library

[4]

M. Black, F. Brard, A. Jepson, W. Newman, E. Saund, G. Socher, and M. Taylor. The digital office: Overview. In Proceedings of the 1998 AAAI Spring Symposium on Intelligent Environments, volume AAAI Technical Report SS-98-02. AAAI, AAAI Press, March 1998.

[5]

P. Chiu, A. Kapuskar, S. Reitmeier, and L. Wilcox. Room with a rear view: Meeting capture in a multimedia conference room. IEEE Multimedia Magazine, 7(4):48-54, Oct-Dec 2000.

Digital Library

[6]

A. J. Diebold. Animal Communication - Techniques of Study and Results of Research, chapter Anthropology of the comparative psychology of communicative behavior. Bloomington: Indiana University Press, 1968.

[7]

A. H. Gee and R. Cipolla. Non-intrusive gaze tracking for human-computer interaction. In Proc. Mechatronics and Machine Vision in Practise, pages 112--117, 1994.

[8]

D. Gopher. The Blackwell dictionary of Cognitive Psychology, chapter Attention, pages 23--28. Basil Blackwell Inc., 1990.

[9]

T. Jebara and A. Pentland. Parametrized structure from motion for 3d adaptive feedback tracking of faces. In Proceedings of Computer Vision and Pattern Recognition, 1997.

Digital Library

[10]

S. R. Langton, R. J. Watt, and V. Bruce. Do the eyes have it? cues to the direction of social attention. Trends in Cognitive Neuroscience, 4(2), 2000.

[11]

M. Mozer. The neural network house: An environment that adapts to its inhibitants. In Intelligent Environments, Papers from the 1998 AAAI Spring Symposium, number Technical Report SS-98-92, pages 110--114. AAAI, AAAI Press, 1998.

[12]

D. Perret and N. Emery. Understanding the intentions of others from visual signals: neurophysiological evidence. Cahiers de Psychologie Cognitive, 13:683-694, 1994.

[13]

R. Stiefelhagen, J. Yang, and A. Waibel. A model-based gaze tracking system. In Proceedings of IEEE International Joint Symposia on Intelligence and Systems, pages 304--310, 1996.

Digital Library

[14]

R. Stiefelhagen, J. Yang, and A. Waibel. Modeling focus of attention for meeting indexing. In Proceedings of ACM Multimedia '99, pages 3--10. ACM, 1999.

Digital Library

[15]

A. Waibel, M. Bett, M. Finke, and R. Stiefelhagen. Meeting browser: Tracking and summarizing meetings. In D. E. M. Penrose, editor, Proceedings of the Broadcast News Transcription and Understanding Workshop, pages 281--286, Lansdowne, Virginia, February. 8-11 1998. DARPA, Morgan Kaufmann.

[16]

J. Yang and A. Waibel. A real-time face tracker. In Proceedings of WACV, pages 142--147, 1996.

Digital Library

Cited By

Tepencelik OWei WLuo MCosman PDey S(2024)Behavioral Intervention for Adults With Autism on Distribution of Attention in Triadic Conversations: A/B-Tested Pre-Post StudyJMIR Formative Research10.2196/553398(e55339)Online publication date: 12-Aug-2024
https://doi.org/10.2196/55339
Rahman SShaikh AMattila HLipping TSari MRaumonen PVirkki J(2023)A Hat-Integrated HCI System for Serious Games–Proof-of-Concept Applications in Focus Detection and Game ControllingGames and Learning Alliance10.1007/978-3-031-49065-1_36(373-382)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1007/978-3-031-49065-1_36
Shynu RSambath R(2021)Environmental Parameters Influencing Perception in the Case of Multimedia CommunicationProceedings of International Conference on Communication and Computational Technologies10.1007/978-981-16-3246-4_47(593-618)Online publication date: 24-Aug-2021
https://doi.org/10.1007/978-981-16-3246-4_47
Show More Cited By

Estimating focus of attention based on gaze and sound
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Tracking head pose and focus of attention with multiple far-field cameras
ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

In this work we present our recent approach on estimating head orientations and foci of attention of multiple people in a smart room, which is equipped with several cameras to monitor the room. In our approach, we estimate each person's head orientation ...
A realtime multimodal system for analyzing group meetings by combining face pose tracking and speaker diarization
ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces

This paper presents a realtime system for analyzing group meetings that uses a novel omnidirectional camera-microphone system. The goal is to automatically discover the visual focus of attention (VFOA), i.e. "who is looking at whom", in addition to ...
Estimating Visual Focus of Attention in Multiparty Meetings using Deep Convolutional Neural Networks
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

Convolutional neural networks (CNNs) are employed to estimate the visual focus of attention (VFoA), also called gaze direction , in multiparty face-to-face meetings on the basis of multimodal nonverbal behaviors including head pose, direction of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PUI '01: Proceedings of the 2001 workshop on Perceptive user interfaces

November 2001

241 pages

ISBN:9781450374736

DOI:10.1145/971478

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

PUI01

PUI01: Workshop on Perceptive User Interfaces

November 15 - 16, 2001

Florida, Orlando, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
633
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)2

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tepencelik OWei WLuo MCosman PDey S(2024)Behavioral Intervention for Adults With Autism on Distribution of Attention in Triadic Conversations: A/B-Tested Pre-Post StudyJMIR Formative Research10.2196/553398(e55339)Online publication date: 12-Aug-2024
https://doi.org/10.2196/55339
Rahman SShaikh AMattila HLipping TSari MRaumonen PVirkki J(2023)A Hat-Integrated HCI System for Serious Games–Proof-of-Concept Applications in Focus Detection and Game ControllingGames and Learning Alliance10.1007/978-3-031-49065-1_36(373-382)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1007/978-3-031-49065-1_36
Shynu RSambath R(2021)Environmental Parameters Influencing Perception in the Case of Multimedia CommunicationProceedings of International Conference on Communication and Computational Technologies10.1007/978-981-16-3246-4_47(593-618)Online publication date: 24-Aug-2021
https://doi.org/10.1007/978-981-16-3246-4_47
Kuzminykh ARintel SBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Low Engagement As a Deliberate Practice of Remote Participants in Video MeetingsExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3383080(1-9)Online publication date: 25-Apr-2020
https://dl.acm.org/doi/10.1145/3334480.3383080
Kuzminykh ARintel SBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Classification of Functional Attention in Video MeetingsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376546(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376546
Porcu SFloris AAnedda MPopescu VFadda MAtzori L(2020)Quality of Experience Eye Gaze Analysis On HbbTV Smart Home Notification System2020 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB)10.1109/BMSB49480.2020.9379794(1-6)Online publication date: 27-Oct-2020
https://doi.org/10.1109/BMSB49480.2020.9379794
Du JDo HSheng W(2020)Human–Robot Collaborative Control in a Virtual-Reality-Based Telepresence SystemInternational Journal of Social Robotics10.1007/s12369-020-00718-w13:6(1295-1306)Online publication date: 9-Nov-2020
https://doi.org/10.1007/s12369-020-00718-w
Reynal MAricò PImbert JHurter CBorghini GDi Flumeri GSciaraffa NDi Florio ATerenzi MFerreira APozzi SBetti VMarucci MBabiloni F(2020)Involving Hearing, Haptics and Kinesthetics into Non-visual Interaction Concepts for an Augmented Remote Tower EnvironmentComputer Vision, Imaging and Computer Graphics Theory and Applications10.1007/978-3-030-41590-7_4(73-100)Online publication date: 20-Feb-2020
https://doi.org/10.1007/978-3-030-41590-7_4
Vinayak RSharma R(2019)When Robots KillInternational Journal of Human Capital and Information Technology Professionals10.4018/IJHCITP.201907010410:3(46-59)Online publication date: Jul-2019
https://doi.org/10.4018/IJHCITP.2019070104
Pan YHachisu TSuzuki K(2017)Computational modeling of head-eye coordination in face-to-face behavior2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC.2017.8122751(1058-1063)Online publication date: Oct-2017
https://doi.org/10.1109/SMC.2017.8122751
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents