Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes
Authors: 
Mansi Sharma, 
Camilo Andrés Martínez Martínez, Benedikt Emanuel Wirth, Antonio Krüger, Philipp MüllerAuthors Info & Claims
Mansi Sharma, Camilo Andrés Martínez Martínez, Benedikt Emanuel Wirth, Antonio Krüger, Philipp MüllerAuthors Info & ClaimsPages 459 - 468
Abstract
Distinguishing target from non-target fixations during visual search is a fundamental building block to understand users’ intended actions and to build effective assistance systems. While prior research indicated the feasibility of classifying target vs. non-target fixations based on eye tracking and electroencephalography (EEG) data, these studies were conducted with explicitly instructed search trajectories, abstract visual stimuli, and disregarded any scene context. This is in stark contrast with the fact that human visual search is largely driven by scene characteristics and raises questions regarding generalizability to more realistic scenarios. To close this gap, we, for the first time, investigate the classification of target vs. non-target fixations during free visual search in realistic scenes. In particular, we conducted a 36-participants user study using a large variety of 140 realistic visual search scenes in two highly relevant application scenarios: searching for icons on desktop backgrounds and finding tools in a cluttered workshop. Our approach based on gaze and EEG features outperforms the previous state-of-the-art approach based on a combination of fixation duration and saccade-related potentials. We perform extensive evaluations to assess the generalizability of our approach across scene types. Our approach significantly advances the ability to distinguish between target and non-target fixations in realistic scenarios, achieving 83.6% accuracy in cross-user evaluations. This substantially outperforms previous methods based on saccade-related potentials, which reached only 56.9% accuracy.
References
[1]
2022. LSL. The Lab Streaming Layer (LSL). (2022). https://github.com/sccn/ labstreaminglayer
[2]
Forrest Sheng Bao, Xin Liu, and Christina Zhang. 2011. PyEEG: An Open Source Python Module for EEG/MEG Feature Extraction. Comp. Int. and Neurosc. (2011), 406391:1–406391:7. https://doi.org/10.1155/2011/406391
[3]
Michael Barz, Sven Stauden, and Daniel Sonntag. 2020. Visual Search Target Inference in Natural Interaction Settings with Machine Learning. In ACM Symposium on Eye Tracking Research and Applications(ETRA ’20 Full Papers). Article 1, 8 pages. https://doi.org/10.1145/3379155.3391314
[4]
Gershon Ben-Shakhar. 2012. Current Research and Potential Applications of the Concealed Information Test: An Overview. Frontiers in Psychology 3 (2012). https://doi.org/10.3389/fpsyg.2012.00342
[5]
Benjamin Blankertz, Ryota Tomioka, Steven Lemm, Motoaki Kawanabe, and Klaus-Robert Muller. 2007. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal processing magazine 25, 1 (2007), 41–56.
[6]
Walter R Boot, Ensar Becic, and Arthur F Kramer. 2009. Stable individual differences in search strategy?: The effect of task demands and motivational factors on scanning strategy in visual search. Journal of Vision 9, 3 (2009), 7–7.
[7]
Ali Borji, Andreas Lennartz, and Marc Pomplun. 2015. What do eyes reveal about the mind?: Algorithmic inference of search targets from fixations. Neurocomputing 149 (2015), 788–799.
[8]
Anne-Marie Brouwer, Maarten A Hogervorst, Bob Oudejans, Anthony J Ries, and Jonathan Touryan. 2017. EEG and eye tracking signatures of target encoding during structured visual search. Frontiers in human neuroscience 11 (2017), 264.
[9]
Anne-Marie Brouwer, Boris Reuderink, Joris Vincent, Marcel AJ van Gerven, and Jan BF van Erp. 2013. Distinguishing between target and nontarget fixations in a visual search task using fixation-related potentials. Journal of vision 13, 3 (2013), 17–17.
[10]
Monica S. Castelhano and John M. Henderson. 2007. Initial scene representations facilitate eye movement guidance in visual search. Journal of Experimental Psychology: Human Perception and Performance 33, 4 (2007), 753–63. https://doi.org/10.1037/0096-1523.33.4.753
[11]
Soumyadip Chatterjee, Saugat Bhattacharyya, Amit Konar, D. N. Tibarewala, Anwesha Khasnobish, and Ramadoss Janarthanan. 2013. Performance Analysis of Multiclass Common Spatial Patterns in Brain-Computer Interface. In Pattern Recognition and Machine Intelligence - 5th International Conference, PReMI 2013. 115–120. https://doi.org/10.1007/978-3-642-45062-4_15
[12]
Arnaud Delorme and Scott Makeig. 2004. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods 134, 1 (2004), 9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
[13]
Olaf Dimigen and Benedikt V Ehinger. 2021. Regression-based analysis of combined EEG and eye-tracking data: Theory and applications. Journal of vision 21, 1 (2021), 3–3.
[14]
Carlos Elmadjian and Carlos H Morimoto. 2021. Gazebar: Exploiting the midas touch in gaze interaction. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7.
[15]
Sarah P. Everett and Michael D. Byrne. 2004. Unintended effects: varying icon spacing changes users’ visual search strategy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI ’04). Association for Computing Machinery, New York, NY, USA, 695–702. https://doi.org/10.1145/985692.985780
[16]
Michael Fleetwood and Michael Byrne. 2006. Modeling the visual search of displays: A revised ACT-R model of icon search based on eye-tracking data. Human-Computer Interaction 21, 2 (2006), 153–197. https://doi.org/10.1207/s15327051hci2102_1
[17]
Charles L. Folk, Roger W. Remington, and James C. Johnston. 1992. Involuntary covert orienting is contingent on attentional control settings. Journal of Experimental Psychology: Human Perception and Performance 18, 4 (1992), 1030–1044. https://doi.org/10.1037//0096-1523.18.4.1030
[18]
Franziska Geringswald, Florian Baumgartner, and Stefan Pollmann. 2012. Simulated loss of foveal vision eliminates visual search advantage in repeated displays. Frontiers in human neuroscience 6 (2012), 134.
[19]
Franziska Geringswald, Anne Herbik, Michael B Hoffmann, and Stefan Pollmann. 2013. Contextual cueing impairment in patients with age-related macular degeneration. Journal of Vision 13, 3 (2013), 28–28.
[20]
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A. Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, and Matti S. Hämäläinen. 2013. MEG and EEG Data Analysis with MNE-Python. Frontiers in Neuroscience 7, 267 (2013), 1–13. https://doi.org/10.3389/fnins.2013.00267
[21]
Fu Guo, Jiahao Chen, Mingming Li, Wei Lyu, and Junjie Zhang. 2022. Effects of visual complexity on user search behavior and satisfaction: an eye-tracking study of mobile news apps. Universal Access in the Information Society 21, 4 (2022), 795–808.
[22]
John K Haas. 2014. A History of the Unity Game Engine. https://api.semanticscholar.org/CorpusID:86824974
[23]
Peter A Hancock, Joseph E Mercado, James Merlo, and Jan BF Van Erp. 2013. Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics 56, 5 (2013), 729–738.
[24]
Andrew Hollingworth. 2009. Two forms of scene memory guide visual search: Memory for scene context and memory for the binding of target object to scene location. Visual Cognition 17, 1-2 (2009), 273–291. https://doi.org/10.1080/13506280802193367
[25]
Tao Jin, Wenrui Wang, Jiamin He, Zhengxin Wu, and Haoran Gu. 2023. Influence mechanism of icon semantics on visual search performance: Evidence from an eye-tracking study. International Journal of Industrial Ergonomics 93 (2023). https://doi.org/10.1016/j.ergon.2022.103402
[26]
Juan E Kamienkowski, Matias J Ison, Rodrigo Quian Quiroga, and Mariano Sigman. 2012. Fixation-related potentials in visual search: A combined EEG and eye tracking study. Journal of vision 12, 7 (2012), 4–4.
[27]
Lisandro N Kaunitz, Juan E Kamienkowski, Alexander Varatharajah, Mariano Sigman, Rodrigo Quian Quiroga, and Matias J Ison. 2014. Looking for a face in the crowd: Fixation-related potentials in an eye-movement visual search task. NeuroImage 89 (2014), 297–305.
[28]
Marius Klug and Klaus Gramann. 2021. Identifying key factors for improving ICA-based decomposition of EEG data in mobile and stationary experiments. European Journal of Neuroscience 54, 12 (2021), 8406–8420. https://doi.org/10.1111/ejn.14992
[29]
Kip Ludwig, Rachel Miriani, Nicholas Langhals, Michael Joseph, David Anderson, and Daryl Kipke. 2009. Using a Common Average Reference to Improve Cortical Neuron Recordings From Microelectrode Arrays. Journal of neurophysiology 101 (03 2009), 1679–89. https://doi.org/10.1152/jn.90989.2008
[30]
Jun Ma, Banghua Yang, Wenzheng Qiu, Yunzhe Li, Shouwei Gao, and Xinxing Xia. 2022. A large EEG dataset for studying cross-session variability in motor imagery brain-computer interface. Scientific Data 9, 1 (2022), 531.
[31]
Pallavi Mohan, Wooi Boon Goh, Chi-Wing Fu, and Sai-Kit Yeung. 2018. DualGaze: Addressing the midas touch problem in gaze mediated VR interaction. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 79–84.
[32]
Philipp Müller, Sander Staal, Mihai Bâce, and Andreas Bulling. 2022. Designing for noticeability: Understanding the impact of visual importance on desktop notifications. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–13.
[33]
Anneli Olsen. 2012. The Tobii IVT Fixation Filter Algorithm description. https://api.semanticscholar.org/CorpusID:52834703
[34]
Stefan Pollmann, Lisa Rosenblum, Stefanie Linnhoff, Eleonora Porracin, Franziska Geringswald, Anne Herbik, Katja Renner, and Michael B Hoffmann. 2020. Preserved contextual cueing in realistic scenes in patients with age-related macular degeneration. Brain Sciences 10, 12 (2020), 941.
[35]
Maurice Rekrut, Mansi Sharma, Matthias Schmitt, Jan Alexandersson, and Antonio Krüger. 2021. Decoding semantic categories from eeg activity in silent speech imagination tasks. In 2021 9th International Winter Conference on Brain-Computer Interface (BCI). IEEE, 1–7.
[36]
Enver Sangineto, Gloria Zen, Elisa Ricci, and Nicu Sebe. 2014. We are not all equal: Personalizing models for facial expression analysis with transductive parameter transfer. In Proceedings of the 22nd ACM international conference on Multimedia. 357–366.
[37]
Hosnieh Sattar, Sabine Müller, Mario Fritz, and Andreas Bulling. 2015. Prediction of search targets from fixations in open-world settings. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 981–990. https://doi.org/10.1109/CVPR.2015.7298700
[38]
Mansi Sharma, Shuang Chen, Philipp Müller, Maurice Rekrut, and Antonio Krüger. 2023. Implicit Search Intent Recognition using EEG and Eye Tracking: Novel Dataset and Cross-User Prediction. In Proceedings of the 25th International Conference on Multimodal Interaction. 345–354.
[39]
Mansi Sharma, Maurice Rekrut, Jan Alexandersson, and Antonio Krüger. 2023. Towards Improving EEG-Based Intent Recognition in Visual Search Tasks. In Neural Information Processing. Springer International Publishing, Cham, 604–615.
[40]
Florian Strohm, Ekta Sood, Sven Mayer, Philipp Müller, Mihai Bâce, and Andreas Bulling. 2021. Neural photofit: Gaze-based mental image reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 245–254.
[41]
Jan Theeuwes. 1992. Perceptual selectivity for color and form. Perception & Psychophysics 51, 6 (1992), 599–606. https://doi.org/10.3758/BF03211656
[42]
A. Torralba, A. Oliva, M. S. Castelhano, and Jeremy M. Henderson. 2006. Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review 113, 4 (2006), 766–86. https://doi.org/10.1037/0033-295X.113.4.766
[43]
Julia Trabulsi, Kian Norouzi, Seidi Suurmets, Mike Storm, and Thomas Zoëga Ramsøy. 2021. Optimizing fixation filters for eye-tracking on small screens. Frontiers in neuroscience (2021), 1257. https://doi.org/10.3389/fnins.2021.578439
[44]
Anne M. Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive Psychology 12, 1 (1980), 97–136. https://doi.org/10.1016/0010-0285(80)90005-5
[45]
Lisa-Marie Vortmann, Simon Ceh, and Felix Putze. 2022. Multimodal eeg and eye tracking feature fusion approaches for attention classification in hybrid bcis. Frontiers in Computer Science 4 (2022), 780580.
[46]
Jeremy M. Wolfe. 2020. Visual search: How do we find what we are looking for?Annual Review of Vision Science 6 (2020), 539–562. https://doi.org/10.1146/annurev-vision-091718-015048
[47]
Jeremy M. Wolfe and Todd S. Horowitz. 2017. Five factors that guide attention in visual search. Nature Human Behaviour 1, 3 (2017). https://doi.org/10.1038/s41562-017-0058
[48]
Jeremy M. Wolfe, Melissa L. Vo, Karla K. Evans, and Michelle R. Greene. 2011. Visual search in scenes involves selective and nonselective pathways. Trends in Cognitive Sciences 15, 2 (2011), 77–84. https://doi.org/10.1016/j.tics.2010.12.001
[49]
Li-Ming Zhao, Xu Yan, and Bao-Liang Lu. 2021. Plug-and-play domain adaptation for cross-subject EEG-based emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 863–870.
Index Terms
- Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes
Recommendations
Implicit Search Intent Recognition using EEG and Eye Tracking: Novel Dataset and Cross-User Prediction
ICMI '23: Proceedings of the 25th International Conference on Multimodal InteractionFor machines to effectively assist humans in challenging visual search tasks, they must differentiate whether a human is simply glancing into a scene (navigational intent) or searching for a target object (informational intent). Previous research ...
Brainwave-Augmented Eye Tracker: High-Frequency SSVEPs Improves Camera-Based Eye Tracking Accuracy
IUI '22: Proceedings of the 27th International Conference on Intelligent User InterfacesIn this work, we leverage neural mechanisms of visual attention to improve the accuracy of a commercial eye tracker through the analysis of electroencephalography (EEG) waves. Gaze targets were rendered in a computer screen with imperceptible flickering ...
Visual search analysis using parametric fixations
AbstractEye movement analysis has been an important area of research. The analysis of eye movements could help in better understanding of human preferences, behavioral pattern and viewing sequences. Eye movement analysis involves scanpath tracking which ...
Comments
Information & Contributors
Information
Published In
November 2024
725 pages
ISBN:9798400704628
DOI:10.1145/3678957
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- German Ministry for Education and Research (BMBF)
Conference
ICMI '24
ICMI '24: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
November 4 - 8, 2024
San Jose, Costa Rica
Acceptance Rates
Overall Acceptance Rate 453 of 1,080 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 178Total Downloads
- Downloads (Last 12 months)178
- Downloads (Last 6 weeks)86
Reflects downloads up to 24 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in