research-article

An Automated High-Level Saliency Predictor for Smart Game Balancing

Authors:

George Alex Koulieris,

George Drettakis,

Douglas Cunningham,

Katerina ManiaAuthors Info & Claims

ACM Transactions on Applied Perception (TAP), Volume 11, Issue 4

Article No.: 17, Pages 1 - 21

https://doi.org/10.1145/2637479

Published: 08 December 2014 Publication History

Abstract

Successfully predicting visual attention can significantly improve many aspects of computer graphics: scene design, interactivity and rendering. Most previous attention models are mainly based on low-level image features, and fail to take into account high-level factors such as scene context, topology, or task. Low-level saliency has previously been combined with task maps, but only for predetermined tasks. Thus, the application of these methods to graphics (e.g., for selective rendering) has not achieved its full potential. In this article, we present the first automated high-level saliency predictor incorporating two hypotheses from perception and cognitive science that can be adapted to different tasks. The first states that a scene is comprised of objects expected to be found in a specific context as well objects out of context which are salient (scene schemata) while the other claims that viewer’s attention is captured by isolated objects (singletons). We propose a new model of attention by extending Eckstein’s Differential Weighting Model. We conducted a formal eye-tracking experiment which confirmed that object saliency guides attention to specific objects in a game scene and determined appropriate parameters for a model. We present a GPU-based system architecture that estimates the probabilities of objects to be attended in real- time. We embedded this tool in a game level editor to automatically adjust game level difficulty based on object saliency, offering a novel way to facilitate game design. We perform a study confirming that game level completion time depends on object topology as predicted by our system.

Supplementary Material

koulieris (koulieris.zip)

Supplemental movie and image files for, An Automated High-Level Saliency Predictor for Smart Game Balancing

Download
57.79 MB

References

[1]

Dan Ariely. 2001. Seeing sets: Representation by statistical properties. Psychological Science 12, 2 (2001), 157--162.

[2]

Edward Awh and Harold Pashler. 2000. Evidence for split attentional foci. Journal of Experimental Psychology: Human Perception and Performance 26, 2 (2000), 834.

[3]

Moshe Bar and Shimon Ullman. 1996. Spatial context in recognition. Perception 25 (1996), 343--352.

[4]

Frederic C. Bartlett. 1932. Remembering: An Experimental and Social Study. Cambridge University Press, Cambridge, UK.

[5]

Mark W. Becker, Harold Pashler, and Jeffrey Lubin. 2007. Object-intrinsic oddities draw early saccades. Journal of Experimental Psychology: Human Perception and Performance 33, 1 (2007), 20.

[6]

Matthias Bernhard, Efstathios Stavrakis, and Michael Wimmer. 2010. An empirical pipeline to derive gaze prediction heuristics for 3D action games. ACM Transactions on Applied Perception (TAP) 8, 1 (2010), 4.

Digital Library

[7]

M. Bernhard, L. Zhang, and M. Wimmer. 2011. Manipulating attention in computer games. In Proceedings of the 2011 IEEE 10th IVMSP Workshop. IEEE, 153--158.

[8]

Ali Borji and Laurent Itti. 2013. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013).

Digital Library

[9]

William F. Brewer and James C. Treyens. 1981. Role of schemata in memory for places. Cognitive Psychology 13, 2 (1981), 207--230.

[10]

Kirsten Cater, Alan Chalmers, and Greg Ward. 2003. Detail to attention: Exploiting visual tasks for selective rendering. In Proceedings of the 14th Eurographics Workshop on Rendering. Eurographics Association, 270--280.

Digital Library

[11]

Douglas W. Cunningham and Christian Wallraven. 2011. Experimental Design: From User Studies to Psychophysics. A.K. Peters.

Digital Library

[12]

Mihaly Czikszentmihalyi. 1990. Flow: The Psychology of Optimal Experience. HarperCollins, New York.

[13]

Heather Desurvire, Martin Caplan, and Jozsef A. Toth. 2004. Using heuristics to evaluate the playability of games. In CHI’04 Extended Abstracts on Human Factors in Computing Systems. ACM, 1509--1512.

Digital Library

[14]

Miguel P. Eckstein. 1998. The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing. Psychological Science 9, 2 (1998), 111--118.

[15]

Miguel P. Eckstein, Barbara A. Drescher, and Steven S. Shimozaki. 2006. Attentional cues in real scenes, saccadic targeting, and Bayesian priors. Psychological Science 17, 11 (2006), 973--980.

[16]

Miguel P. Eckstein, Steven S. Shimozaki, and Craig K. Abbey. 2002. The footprints of visual attention in the Posner cueing paradigm revealed by classification images. Journal of Vision 2, 1 (2002).

[17]

Wolfgang Einhäuser, Merrielle Spain, and Pietro Perona. 2008. Objects predict fixations better than early saliency. Journal of Vision 8, 14 (2008).

[18]

Magy Seif El-Nasr and Su Yan. 2006. Visual attention in 3D video games. In Proceedings of the 2006 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology. ACM, 22.

Digital Library

[19]

John Feil and Marc Scattergood. 2005. Beginning Game Level Design. Course Technology PTR.

[20]

Simone Frintrop, Erich Rome, and Henrik I. Christensen. 2010. Computational visual attention systems and their cognitive foundations: A survey. ACM Transactions on Applied Perception (TAP) 7, 1 (2010), 6.

Digital Library

[21]

David Marvin Green and John A. Swets. 1966. Signal Detection Theory and Psychophysics. Vol. 1. Wiley, New York.

[22]

Helena Grillon and Daniel Thalmann. 2009. Simulating gaze attention behaviors for crowds. Computer Animation and Virtual Worlds 20, 2--3 (2009), 111--119.

Digital Library

[23]

John M. Henderson and Andrew Hollingworth. 1999. High-level scene perception. Annual Review of Psychology 50, 1 (1999), 243--271.

[24]

John M. Henderson, Phillip A. Weeks Jr, and Andrew Hollingworth. 1999. The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology: Human Perception and Performance 25, 1 (1999), 210.

[25]

Sebastien Hillaire, Anatole Lécuyer, Tony Regia-Corte, Rémi Cozot, Jérome Royan, and Gaspard Breton. 2010. A real-time visual attention model for predicting gaze point during first-person exploration of virtual environments. In Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology. ACM, 191--198.

Digital Library

[26]

Alex D. Hwang, Hsueh-Cheng Wang, and Marc Pomplun. 2011. Semantic guidance of eye movements in real-world scenes. Vision Research 51, 10 (2011), 1192--1205.

[27]

Laurent Itti and Christof Koch. 2001. Computational modelling of visual attention. Nature Reviews Neuroscience 2, 3 (2001), 194--203.

[28]

Daniel Johnson and Janet Wiles. 2003. Effective affective user interface design in games. Ergonomics 46, 13--14 (2003), 1332--1345.

[29]

Edward Ju and Christian Wagner. 1997. Personal computer adventure games: Their structure, principles, and applicability for training. ACM SIGMIS Database 28, 2 (1997), 78--92.

Digital Library

[30]

Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. 2009. Learning to predict where humans look. In Proceedings of the 2009 IEEE 12th international conference on Computer Graphics. IEEE, 2106--2113.

[31]

Christof Koch and Shimon Ullman. 1987. Shifts in selective visual attention: Towards the underlying neural circuitry. In Matters of Intelligence. Springer, 115--141.

[32]

George Alex Koulieris, George Drettakis, Douglas Cunningham, and Katerina Mania. 2014. C-LOD: Context-aware material level-of-detail applied to mobile graphics. In Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering 2014, Lyon, France) 33, 4 (2014).

Digital Library

[33]

Sungkil Lee, Gerard Jounghyun Kim, and Seungmoon Choi. 2009. Real-time tracking of visually attended objects in virtual environments and its application to lod. IEEE Transactions on Visualization and Computer Graphics 15, 1 (2009), 6--19.

Digital Library

[34]

Peter Longhurst, Kurt Debattista, and Alan Chalmers. 2006. A GPU based saliency map for high-fidelity selective rendering. In Proceedings of the 4th International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa. ACM, 21--29.

Digital Library

[35]

Katerina Mania, Andrew Robinson, and Karen R. Brandt. 2005. The effect of memory schemas on object recognition in virtual environments. Presence: Teleoperators and Virtual Environments 14, 5 (2005), 606--615.

Digital Library

[36]

Markos Markou and Sameer Singh. 2003. Novelty detection: A review part 2, neural network based approaches. Signal Processing 83, 12 (2003), 2499--2521.

Digital Library

[37]

David Marr. 1982. Vision: A computational investigation into the human representation and processing of visual information, Henry Holt and Co., New York.

Digital Library

[38]

Mathias Moehring, Antje Gloystein, and Ralf Doerner. 2009. Issues with virtual space perception within reaching distance: Mitigating adverse effects on applications using HMDs in the automotive industry. In Proceedings of the 2009 Virtual Reality Conference (VR’09). IEEE, 223--226.

Digital Library

[39]

Nicholaos Mourkoussis, Fiona M. Rivera, Tom Troscianko, Tim Dixon, Rycharde Hawkes, and Katerina Mania. 2010. Quantifying fidelity for virtual environment simulations employing memory schema assumptions. ACM Transactions on Applied Perception (TAP) 8, 1 (2010), 2.

Digital Library

[40]

Ken Nakayama and Gerald H. Silverman. 1986. Serial and parallel processing of visual feature conjunctions. Nature 320, 6059 (1986), 264--265.

[41]

Kathleen M. O’Craven, Paul E. Downing, and Nancy Kanwisher. 1999. fMRI evidence for objects as the units of attentional selection. Nature 401, 6753 (1999), 584--587.

[42]

Oyewole Oyekoya, William Steptoe, and Anthony Steed. 2009. A saliency-based method of simulating visual attention in virtual scenes. In Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology. ACM, 199--206.

Digital Library

[43]

Randy J. Pagulayan, Kevin Keeker, Dennis Wixon, Ramon L. Romero, and Thomas Fuller. 2003. User-centered design in games. In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. 883--906.

Digital Library

[44]

Stephen E. Palmer. 1999. Vision Science: Photons to Phenomenology. MIT Press.

[45]

Keith Rayner. 2009. Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology 62, 8 (2009), 1457--1506.

[46]

Ronald A. Rensink. 2000. The dynamic representation of scenes. Visual Cognition 7, 1--3 (2000), 17--42.

[47]

Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications. ACM, 71--78.

Digital Library

[48]

Thomas F. Shipley and Philip J. Kellman. 2001. From Fragments to Objects: Segmentation and Grouping in Vision. Vol. 130. Elsevier.

[49]

Peter N. Steinmetz, A. Roy, P. J. Fitzgerald, S. S. Hsiao, K. O. Johnson, and E. Niebur. 2000. Attention modulates synchronized neuronal firing in primate somatosensory cortex. Nature 404, 6774 (2000), 187--190.

[50]

Veronica Sundstedt, Matthias Bernhard, Efstathios Stavrakis, Erik Reinhard, and Michael Wimmer. 2013. Visual attention and gaze behavior in games: An object-based approach. In Game Analytics. Springer, 543--583.

[51]

Veronica Sundstedt, Alan Chalmers, Kirsten Cater, and Kurt Debattista. 2004. Top-down visual attention for efficient rendering of task related scenes. In VMV. 209--216.

[52]

Veronica Sundstedt, Kurt Debattista, Peter Longhurst, Alan Chalmers, and Tom Troscianko. 2005. Visual attention for efficient high-fidelity graphics. In Proceedings of the 21st Spring Conference on Computer Graphics. ACM, 169--175.

Digital Library

[53]

Veronica Sundstedt, Efstathios Stavrakis, Michael Wimmer, and Erik Reinhard. 2008. A psychophysical study of fixation behavior in a computer game. In Proceedings of the 5th Symposium on Applied Perception in Graphics and Visualization. ACM, 43--50.

Digital Library

[54]

Penelope Sweetser and Peta Wyeth. 2005. GameFlow: A model for evaluating player enjoyment in games. Computers in Entertainment (CIE) 3, 3 (2005), 3--3.

Digital Library

[55]

Jan Theeuwes. 2010. Top--down and bottom--up control of visual selection. Acta Psychologica 135, 2 (2010), 77--99.

[56]

Jan Theeuwes and Richard Godijn. 2002. Irrelevant singletons capture attention: Evidence from inhibition of return. Perception & Psychophysics 64, 5 (2002), 764--770.

[57]

David J. Tolhurst, J. Anthony Movshon, and A. F. Dean. 1983. The statistical reliability of signals in single neurons in cat and monkey visual cortex. Vision Research 23, 8 (1983), 775--785.

[58]

Anne M. Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive Psychology 12, 1 (1980), 97--136.

[59]

Dirk Walther and Christof Koch. 2006. Modeling attention to salient proto-objects. Neural Networks 19, 9 (2006), 1395--1407.

Digital Library

[60]

Jeremy M. Wolfe. 1994. Guided search 2.0 a revised model of visual search. Psychonomic Bulletin & Review 1, 2 (1994), 202--238.

[61]

Alfred L. Yarbus, Basil Haigh, and Lorrin A. Rigss. 1967. Eye Movements and Vision. Vol. 2. Plenum Press, New York.

[62]

Alexandros Zotos, Katerina Mania, and Nicholaos Mourkoussis. 2009. A schema-based selective rendering framework. In Proceedings of the 6th Symposium on Applied Perception in Graphics and Visualization. ACM, 85--92.

Digital Library

Cited By

Javerliat CVillenave SRaimbaud PLavoué G(2024)PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR ExperiencesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210730:5(2087-2097)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372107
Berkman M(2024)Eye Tracking in Virtual RealityEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_170(681-688)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_170
Duchowski A(2022)Gaze-based interactionComputers and Graphics10.1016/j.cag.2018.04.00273:C(59-69)Online publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1016/j.cag.2018.04.002
Show More Cited By

Index Terms

An Automated High-Level Saliency Predictor for Smart Game Balancing
1. Computing methodologies
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Object-level saliency detection with color attributes

Recently, saliency detection has been becoming a popular topic in computer vision. In this paper we propose an object-level saliency detection algorithm which explicitly explores bottom-up visual attention and objectness cues. Firstly, some category-...
An improved illumination model for shaded display

To accurately render a two-dimensional image of a three-dimensional scene, global illumination information that affects the intensity of each pixel of the image must be known at the time the intensity is calculated. In a simplified form, this ...
An improved illumination model for shaded display
SIGGRAPH '05: ACM SIGGRAPH 2005 Courses

To accurately render a two-dimensional image of a three-dimensional scene, global illumination information that affects the intensity of each pixel of the image must be known at the time the intensity is calculated. In a simplified form, this ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Applied Perception

ACM Transactions on Applied Perception Volume 11, Issue 4

January 2015

132 pages

ISSN:1544-3558

EISSN:1544-3965

DOI:10.1145/2695584

Editors:
Roland Fleming
University of Giessen, Germany
,
Carol O'Sullivan
Trinity College Dublin, Ireland

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2014

Accepted: 01 June 2014

Revised: 01 May 2014

Received: 01 December 2013

Published in TAP Volume 11, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

European Social Fund
Seventh Framework Programme
Greek national funds through the Operational Program “Education and Lifelong Learning”

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
317
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Javerliat CVillenave SRaimbaud PLavoué G(2024)PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR ExperiencesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210730:5(2087-2097)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372107
Berkman M(2024)Eye Tracking in Virtual RealityEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_170(681-688)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_170
Duchowski A(2022)Gaze-based interactionComputers and Graphics10.1016/j.cag.2018.04.00273:C(59-69)Online publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1016/j.cag.2018.04.002
Duchowski A(2020)Eye-based interaction in graphical systemsACM SIGGRAPH 2020 Courses10.1145/3388769.3407492(1-246)Online publication date: 17-Aug-2020
https://dl.acm.org/doi/10.1145/3388769.3407492
Koulieris GAkşit KStengel MMantiuk RMania KRichardt C(2019)Near‐Eye Display and Tracking Technologies for Virtual and Augmented RealityComputer Graphics Forum10.1111/cgf.1365438:2(493-519)Online publication date: 7-Jun-2019
https://doi.org/10.1111/cgf.13654
Lavoué GCordier FSeo HLarabi M(2018)Visual Attention for Rendered 3D ShapesComputer Graphics Forum10.1111/cgf.1335337:2(191-203)Online publication date: 22-May-2018
https://doi.org/10.1111/cgf.13353
Berkman M(2018)Eye Tracking in Virtual RealityEncyclopedia of Computer Graphics and Games10.1007/978-3-319-08234-9_170-1(1-8)Online publication date: 24-Nov-2018
https://doi.org/10.1007/978-3-319-08234-9_170-1
Krishna OAizawa KCunningham DBreuß MJoerg SInterrante V(2017)Age-adapted saliency model with depth biasProceedings of the ACM Symposium on Applied Perception10.1145/3119881.3119885(1-8)Online publication date: 16-Sep-2017
https://dl.acm.org/doi/10.1145/3119881.3119885
Weier MStengel MRoth TDidyk PEisemann EEisemann MGrogorick SHinkenjann AKruijff EMagnor MMyszkowski KSlusallek P(2017)Perception-driven Accelerated RenderingComputer Graphics Forum10.1111/cgf.1315036:2(611-643)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1111/cgf.13150
Ahmadi HTootaghaj SMowlaei SHashemi MShirmohammadi STimmerer C(2016)GSET somiProceedings of the 7th International Conference on Multimedia Systems10.1145/2910017.2910616(1-6)Online publication date: 10-May-2016
https://dl.acm.org/doi/10.1145/2910017.2910616
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents