research-article

Multi-mode saliency dynamics model for analyzing gaze and attention

Authors:

Hiroaki Kawashima,

Takashi MatsuyamaAuthors Info & Claims

ETRA '12: Proceedings of the Symposium on Eye Tracking Research and Applications

Pages 115 - 122

https://doi.org/10.1145/2168556.2168574

Published: 28 March 2012 Publication History

Abstract

We present a method to analyze a relationship between eye movements and saliency dynamics in videos for estimating attentive states of users while they watch the videos. The multi-mode saliency-dynamics model (MMSDM) is introduced to segment spatio-temporal patterns of the saliency dynamics into multiple sequences of primitive modes underlying the saliency patterns. The MMSDM enables us to describe the relationship by the local saliency dynamics around gaze points, which is modeled by a set of distances between gaze points and salient regions characterized by the extracted modes. Experimental results show the effectiveness of the proposed model to classify the attentive states of users by learning the statistical difference of the local saliency dynamics on gaze-paths at each level of attentiveness.

References

[1]

Brandherm, B., Prendinger, H., and Ishizuka, M. 2007. Interest estimation based on dynamic bayesian networks for visual attentive presentation agents. In Proc. of ICMI, 346--349.

Digital Library

[2]

Bregler, C. 1997. Learning and recognizing human dynamics in video sequences. In Proc. of CVPR, 568--574.

Digital Library

[3]

Calder, A., Lawrence, A., Keane, J., Scott, S., Owen, A., Christoffels, I., and Young, A. 2002. Reading the mind from eye gaze. Neuropsychologia 40, 8, 1129--1138.

[4]

Dorr, M., Martinetz, T., Gegenfurtner, K., and Barth, E. 2010. Variability of eye movements when viewing dynamic natural scenes. Journal of vision 10, 10, 1--17.

[5]

Doshi, A., and Trivedi, M. M. 2010. Attention estimation by simultaneous observation of viewer and view. In Proc. of CVPR Workshop, 21--27.

[6]

Fletcher, L., and Zelinsky, A. 2009. Driver inattention detection based on eye gaze-road event correlation. International Journal of Robotics Research 28, 6, 774--801.

Digital Library

[7]

Goldstein, R. B., Woods, R. L., and Peli, E. 2007. Where people look when watching movies: do all viewers look at the same place? Computers in Biology and Medicine 37, 7, 957--64.

Digital Library

[8]

Hirayama, T., Dodane, J.-B., Kawashima, H., and Matsuyama, T. 2010. Estimates of user interest using timing structures between proactive content-display updates and eye movements. IEICE Trans. on Information and Systems E-93D, 6, 1470--1478.

[9]

Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. on PAMI 20, 11, 1254--1259.

Digital Library

[10]

Ji, Q., Lan, P., and Looney, C. 2006. A probabilistic framework for modeling and real-time monitoring human fatigue. IEEE Trans. on Systems, Man and Cybernetics, Part A: Systems and Humans 36, 5, 862--875.

Digital Library

[11]

Just, M., and Carpenter, P. 1976. Eye fixations and cognitive processes. Cognitive Psychology 480, 441--480.

[12]

Kahneman, D. 1973. Attention and effort. Prentice Hall.

[13]

Kawashima, H., and Matsuyama, T. 2005. Multiphase learning for an interval-based hybrid dynamical system. IEICE Trans. on Fundamentals E88-A, 11, 3022--3035.

Digital Library

[14]

Li, Y., Wang, T., and Shum, H.-Y. 2002. Motion texture: a two-level statistical model for character motion synthesis. ACM Trans. on Graphics 21, 3, 465--472.

Digital Library

[15]

Munn, S. M., Stefano, L., and Pelz, J. B. 2008. Fixation-identification in dynamic scenes. In Proc. of APGV, 33--42.

Digital Library

[16]

Nakano, Y. I., and Ishii, R. 2010. Estimating user's engagement from eye-gaze behaviors in human-agent conversations. In Proc. of IUI, 139--148.

Digital Library

[17]

North, B., Blake, A., Isard, M., and Rittscher, J. 2000. Learning and classification of complex dynamics. IEEE Trans. on PAMI 22, 9, 1016--1034.

Digital Library

[18]

Pavlovic, V., Rehg, J. M., and Maccormick, J. 2000. Learning switching linear models of human motion. In Proc. of NIPS, 981--987.

[19]

Qvarfordt, P., and Zhai, S. 2005. Conversing with the user based on eye-gaze patterns. In Proc. of CHI, 221--230.

Digital Library

[20]

Schneider, W., and Shiffrin, R. M. 1977. Controlled and automatic human information processing: I. detection, search, and attention. Psychological Review 84, 1--66.

[21]

Walther, D., and Koch, C. 2006. Modeling attention to salient proto-objects. Neural Networks: the Official Journal of the International Neural Network Society 19, 9, 1395--1407.

Digital Library

[22]

Wildes, R. P. 1998. A measure of motion salience for surveillance applications. In Proc. of ICIP, 183--187.

Cited By

Wong ABryck RBaker RHutt SMills CHilliger IKhosravi HRienties BDawson S(2023)Using a Webcam Based Eye-tracker to Understand Students’ Thought Patterns and Reading Behaviors in Neurodivergent ClassroomsLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576115(453-463)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576115
Hutt SD'Mello S(2022)Evaluating Calibration-free Webcam-based Eye Tracking for Gaze-based User ModelingProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556580(224-235)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556580
Hutt SStewart AGregg JMattingly SD'Mello S(2022)Feasibility of Longitudinal Eye-Gaze Tracking in the WorkplaceProceedings of the ACM on Human-Computer Interaction10.1145/35308896:ETRA(1-21)Online publication date: 13-May-2022
https://dl.acm.org/doi/10.1145/3530889
Show More Cited By

Index Terms

Multi-mode saliency dynamics model for analyzing gaze and attention
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

A neural network implementation of a saliency map model

The saliency map model proposed by Itti and Koch [Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 1489-1506] has been a popular model for explaining the guidance of ...
Perceptually guided high-fidelity rendering exploiting movement bias in visual attention

A major obstacle for real-time rendering of high-fidelity graphics is computational complexity. A key point to consider in the pursuit of “realism in real time” in computer graphics is that the Human Visual System (HVS) is a fundamental part of the ...
A saliency-based bottom-up visual attention model for dynamic scenes analysis

This work proposes a model of visual bottom-up attention for dynamic scene analysis. Our work adds motion saliency calculations to a neural network model with realistic temporal dynamics [(e.g., building motion salience on top of De Brecht and Saiki ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '12: Proceedings of the Symposium on Eye Tracking Research and Applications

March 2012

420 pages

ISBN:9781450312219

DOI:10.1145/2168556

Conference Chairs:
Carlos H. Morimoto
University of São Paulo, Brazil
,
Howell Istance
Montfort University, UK
,
Editor:
Stephen N. Spencer
University of Washington
,
Program Chairs:
Jeffrey B. Mulligan
NASA
,
Pernilla Qvarfordt
FX Palo Alto

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 March 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ETRA '12

Sponsor:

ETRA '12: Eye Tracking Research and Applications

March 28 - 30, 2012

California, Santa Barbara

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
343
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wong ABryck RBaker RHutt SMills CHilliger IKhosravi HRienties BDawson S(2023)Using a Webcam Based Eye-tracker to Understand Students’ Thought Patterns and Reading Behaviors in Neurodivergent ClassroomsLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576115(453-463)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576115
Hutt SD'Mello S(2022)Evaluating Calibration-free Webcam-based Eye Tracking for Gaze-based User ModelingProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556580(224-235)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556580
Hutt SStewart AGregg JMattingly SD'Mello S(2022)Feasibility of Longitudinal Eye-Gaze Tracking in the WorkplaceProceedings of the ACM on Human-Computer Interaction10.1145/35308896:ETRA(1-21)Online publication date: 13-May-2022
https://dl.acm.org/doi/10.1145/3530889
Hutt SKrasich KMills CBosch NWhite SBrockmole JD’Mello S(2022)Automated gaze-based mind wandering detection during computerized learning in classroomsUser Modeling and User-Adapted Interaction10.1007/s11257-019-09228-529:4(821-867)Online publication date: 11-Mar-2022
https://dl.acm.org/doi/10.1007/s11257-019-09228-5
D’Mello S(2018)Gaze-Based Attention-Aware Cyberlearning TechnologiesMind, Brain and Technology10.1007/978-3-030-02631-8_6(87-105)Online publication date: 20-Dec-2018
https://doi.org/10.1007/978-3-030-02631-8_6
D'Mello SKopp KBixler RBosch NKaye JDruin ALampe CMorris DHourcade J(2016)Attending to AttentionProceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems10.1145/2851581.2892329(1661-1669)Online publication date: 7-May-2016
https://dl.acm.org/doi/10.1145/2851581.2892329
Bixler RD'mello S(2016)Automatic gaze-based user-independent detection of mind wandering during computerized readingUser Modeling and User-Adapted Interaction10.1007/s11257-015-9167-126:1(33-68)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1007/s11257-015-9167-1
Bixler RBlanchard NGarrison LD'Mello SZhang ZCohen PBohus DHoraud RMeng H(2015)Automatic Detection of Mind Wandering During Reading Using Gaze and PhysiologyProceedings of the 2015 ACM on International Conference on Multimodal Interaction10.1145/2818346.2820742(299-306)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1145/2818346.2820742
Schaffer EKawashima HMatsuyama T(2015)Estimation of browsing states in consumer decision processes from eye movements2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)10.1109/ACPR.2015.7486543(448-453)Online publication date: Nov-2015
https://doi.org/10.1109/ACPR.2015.7486543
Bixler RD’Mello S(2015)Automatic Gaze-Based Detection of Mind Wandering with Metacognitive AwarenessUser Modeling, Adaptation and Personalization10.1007/978-3-319-20267-9_3(31-43)Online publication date: 11-Jun-2015
https://doi.org/10.1007/978-3-319-20267-9_3
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents