CSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze Tracking
Article No.: 6, Pages 1 - 7
Abstract
This paper presents a novel Contrastive Self-Attention Convolutional Neural Network (CSA-CNN) model with enhanced Difficulty Aware (DA) loss function to improve the segmentation of pupils in eye images. The incorporation of transformer-style self-attention and Difficulty-Aware loss in a UNET-style architecture allows for robust feature representation and promotes shape alignment. The novel model was trained on two public databases (LPW and RIT-Eyes) and evaluated on two other public datasets (ExCuSe and ElSe). When compared with seven state-of-the-art pupil center detection methods, the CSA-CNN showed improvement of over 6% in pupil center detection accuracy (detection within 5 pixels of the labeled center) and more than 9% in Intersection Over Union (IOU) accuracy, compared to the best performer among the other seven methods. Furthermore, when the CSA-CNN model was integrated into a glint-based eye tracking system that uses learning based methods to detect pupil-center, we saw a 25% improvement in gaze accuracy.
References
[1]
Michael Abrash. 2021. Creating the future: Augmented reality, the next human-machine interface. In 2021 IEEE International Electron Devices Meeting (IEDM). IEEE, 1–2.
[2]
Cuneyt Akinlar, Hatice Kubra Kucukkartal, and Cihan Topal. 2022. Accurate cnn-based pupil segmentation with an ellipse fit error regularization term. Expert Systems with Applications 188 (2022), 116004.
[3]
Braiden Brousseau, Jonathan Rose, and Moshe Eizenman. 2020. Hybrid eye-tracking on a smartphone with CNN feature extraction and an infrared 3D model. Sensors 20, 2 (2020), 543.
[4]
Sushil Chandra, Greeshma Sharma, Saloni Malhotra, Devendra Jha, and Alok Prakash Mittal. 2015. Eye tracking based human computer interaction: Applications and their uses. In 2015 International Conference on Man and Machine Interfacing (MAMI). IEEE, 1–5.
[5]
Aayush K. Chaudhary, Rakshit Kothari, Manoj Acharya, Shusil Dangi, Nitinraj Nair, Reynold Bailey, Christopher Kanan, Gabriel Diaz, and Jeff B. Pelz. 2019. RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2019.00568
[6]
Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu. 2021. Appearance-based gaze estimation with deep learning: A review and benchmark. arXiv preprint arXiv:2104.12668 (2021).
[7]
Soumil Chugh. 2020. An Eye Tracking System for a Virtual Reality Headset. University of Toronto (Canada).
[8]
Soumil Chugh, Braiden Brousseau, Jonathan Rose, and Moshe Eizenman. 2021. Detection and Correspondence Matching of Corneal Reflections for Eye Tracking Using Deep Learning. 2020 25th International Conference on Pattern Recognition (ICPR) (2021), 2210–2217.
[9]
Kai Dierkes, Moritz Kassner, and Andreas Bulling. 2018. A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications. 1–9.
[10]
Wolfgang Fuhl, Hong Gao, and Enkelejda Kasneci. 2020. Tiny convolution, decision tree, and binary neuronal networks for robust and real time pupil outline estimation. In ACM symposium on eye tracking research and applications. 1–5.
[11]
Wolfgang Fuhl, David Geisler, Thiago Santini, Tobias Appel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2018. CBF: Circular binary features for robust and real-time pupil center detection. In Proceedings of the 2018 ACM symposium on eye tracking research & applications. 1–6.
[12]
Wolfgang Fuhl, Thomas C. Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015a. ExCuSe: Robust Pupil Detection in Real-World Scenarios. In International Conference on Computer Analysis of Images and Patterns.
[13]
Wolfgang Fuhl, Thiago C. Santini, Thomas Kuebler, and Enkelejda Kasneci. 2015b. ElSe: Ellipse Selection for Robust Pupil Detection in Real-World Environments. arxiv:1511.06575 [cs.CV]
[14]
Wolfgang Fuhl, Johannes Schneider, and Enkelejda Kasneci. 2021. 1000 pupil segmentations in a second using haar like features and statistical learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3466–3476.
[15]
Swathi Gowroju, Sandeep Kumar, 2020. Robust deep learning technique: U-Net architecture for pupil segmentation. In 2020 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON). IEEE, 0609–0613.
[16]
Elias Daniel Guestrin and Moshe Eizenman. 2006. General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE transactions on bio-medical engineering 53, 6 (June 2006), 1124—1133. https://doi.org/10.1109/tbme.2005.863952
[17]
Sang Yoon Han, Hyuk Jin Kwon, Yoonsik Kim, and Nam Ik Cho. 2020. Noise-robust pupil center detection through CNN-based segmentation with shape-prior loss. IEEE Access 8 (2020), 64739–64749.
[18]
Benedikt Hosp, Shahram Eivazi, Maximilian Maurer, Wolfgang Fuhl, David Geisler, and Enkelejda Kasneci. 2020. RemoteEye: An open-source high-speed remote eye tracker: Implementation insights of a pupil-and glint-detection algorithm for high-speed remote eye tracking. Behavior research methods 52 (2020), 1387–1401.
[19]
Rakshit S Kothari, Aayush K Chaudhary, Reynold J Bailey, Jeff B Pelz, and Gabriel J Diaz. 2021. Ellseg: An ellipse segmentation framework for robust gaze tracking. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2757–2767.
[20]
Yu-Cheng Liu, Daniel Stanley Tan, Jyh-Cheng Chen, Wen-Huang Cheng, and Kai-Lung Hua. 2019. Segmenting hepatic lesions using residual attention U-Net with an adaptive weighted dice loss. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 3322–3326.
[21]
Nitinraj Nair, Rakshit Kothari, Aayush K Chaudhary, Zhizhuo Yang, Gabriel J Diaz, Jeff B Pelz, and Reynold J Bailey. 2020. Rit-eyes: Rendering of near-eye images for eye-tracking applications. In ACM Symposium on Applied Perception 2020. 1–9.
[22]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 234–241.
[23]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510–4520.
[24]
Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. 2018. PuRe: Robust pupil detection for real-time pervasive eye tracking. Computer Vision and Image Understanding 170 (2018), 40–50. https://doi.org/10.1016/j.cviu.2018.02.002
[25]
Thiago Santini, Diederick C Niehorster, and Enkelejda Kasneci. 2019. Get a grip: Slippage-robust and glint-free gaze estimation for real-time pervasive head-mounted eye tracking. In Proceedings of the 11th ACM symposium on eye tracking research & applications. 1–10.
[26]
A Sathesh. 2019. Typing Eyes: A Human Computer Interface Technology. Journal of Electronics and Informatics 1, 2 (2019), 80–88.
[27]
A Swathi, Sandeep Kumar, 2021. Review on pupil segmentation using CNN-region of interest. In Intelligent communication and automation systems. CRC Press, 157–168.
[28]
Marc Tonsen, Xucong Zhang, Yusuke Sugano, and Andreas Bulling. 2016. Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments. In Proceedings of the ninth biennial ACM symposium on eye tracking research & applications. 139–142.
[29]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[30]
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. 2021. Exploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7303–7313.
[31]
Yuk-Hoi Yiu, Moustafa Aboulatta, Theresa Raiser, Leoni Ophey, Virginia L Flanagin, Peter Zu Eulenburg, and Seyed-Ahmad Ahmadi. 2019. DeepVOG: Open-source pupil segmentation and gaze estimation in neuroscience using deep learning. Journal of neuroscience methods 324 (2019), 108307.
[32]
Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems 31 (2018).
Index Terms
- CSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze Tracking
Recommendations
Eye and gaze tracking for interactive graphic display
This paper describes a computer vision system based on active IR illumination for real-time gaze tracking for interactive graphic display. Unlike most of the existing gaze tracking techniques, which often require assuming a static head to work well and ...
A non-contact device for tracking gaze in a human computer interface
Special issue on eye detection and trackingThis paper presents a novel design for a non-contact eye detection and gaze tracking device. It uses two cameras to maintain real-time tracking of a person's eye in the presence of head motion. Image analysis techniques are used to obtain accurate ...
Eye and gaze tracking for interactive graphic display
SMARTGRAPH '02: Proceedings of the 2nd international symposium on Smart graphicsThis paper describes preliminary results we have obtained in developing a computer vision system based on active IR illumination for real time gaze tracking for interactive graphic display. Unlike most of the existing gaze tracking techniques, which ...
Comments
Information & Contributors
Information
Published In
June 2024
525 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ETRA '24: The 2024 Symposium on Eye Tracking Research and Applications
June 4 - 7, 2024
Glasgow, United Kingdom
Acceptance Rates
Overall Acceptance Rate 69 of 137 submissions, 50%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 134Total Downloads
- Downloads (Last 12 months)134
- Downloads (Last 6 weeks)14
Reflects downloads up to 27 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format