Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

EllSeg-Gen, towards Domain Generalization for Head-Mounted Eyetracking

Published: 13 May 2022 Publication History

Abstract

The study of human gaze behavior in natural contexts requires algorithms for gaze estimation that are robust to a wide range of imaging conditions. However, algorithms often fail to identify features such as the iris and pupil centroid in the presence of reflective artifacts and occlusions. Previous work has shown that convolutional networks excel at extracting gaze features despite the presence of such artifacts. However, these networks often perform poorly on data unseen during training. This work follows the intuition that jointly training a convolutional network with multiple datasets learns a generalized representation of eye parts. We compare the performance of a single model trained with multiple datasets against a pool of models trained on individual datasets. Results indicate that models tested on datasets in which eye images exhibit higher appearance variability benefit from multiset training. In contrast, dataset-specific models generalize better onto eye images with lower appearance variability.

Supplementary Material

PDF File (v6etra139aux.pdf)
MP4 File (1063_full_talk_kothari_rakshit - Rakshit Kothari.mp4)
Conference Presentation (ETRA Long Papers) of title={EllSeg-Gen, towards Domain Generalization for Head-Mounted Eyetracking}; authors={Rakshit S. Kothari, Reynold J. Bailey, Christopher Kanan, Jeff B. Pelz, and Gabriel J. Diaz}; doi=10.1145/3530880

References

[1]
1] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018. Metareg: Towards domain generalization using meta-regularization. Advances in Neural Information Processing Systems 2018-Decem, NeurIPS (2018), 998--1008.
[2]
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from di#erent domains. Machine learning 79, 1--2 (2010), 151--175.
[3]
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. 2006. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22, 14 (2006), e49--e57.
[4]
Aayush K. Chaudhary, Rakshit Kothari, Manoj Acharya, Shusil Dangi, Nitinraj Nair, Reynold Bailey, Christopher Kanan, Gabriel DIaz, and Je# B. Pelz. 2019. RITnet: Real-time semantic segmentation of the eye for gaze tracking. In Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019. IEEE, 3698--3702. https: //doi.org/10.1109/ICCVW.2019.00568 arXiv:1910.00694
[5]
Kai Dierkes, Moritz Kassner, and Andreas Bulling. 2018. A novel approach to single camera, glint-free 3D eye model "tting including corneal refraction. Eye Tracking Research and Applications Symposium (ETRA) June (2018). https://doi.org/10.1145/3204493.3204525
[6]
Kai Dierkes, Moritz Kassner, and Andreas Bulling. 2019. A fast approach to refraction-aware eye-model "tting and gaze prediction. June (2019), 1--9. https://doi.org/10.1145/3314111.3319819
[7]
Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B Blaschko. 2020. Optimization for medical image segmentation: theory and practice when evaluating with Dice score or Jaccard index. IEEE Transactions on Medical Imaging 39, 11 (2020), 3679--3690.
[8]
Shaharam Eivazi, Thiago Santini, Alireza Keshavarzi, Thomas Kübler, and Andrea Mazzei. 2019. Improving real-time CNN-based pupil detection through domain-speci"c data augmentation. (2019), 1--6. https://doi.org/10.1145/3314111. 3319914
[9]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. PMLR, 1126--1135.
[10]
Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015. Excuse: Robust pupil detection in real-world scenarios. In International conference on computer analysis of images and patterns. Springer, 39--51.
[11]
Wolfgang Fuhl, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2019. 500,000 Images Closer to Eyelid and Pupil Segmentation. Lecture Notes in Computer Science, Vol. 11678 LNCS. Springer International Publishing, Cham. 336--347 pages. https://doi.org/10.1007/978--3-030--29888--3{_}27
[12]
Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2017. PupilNet v2.0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection. (2017). http://arxiv.org/abs/1711.00112
[13]
Wolfgang Fuhl, Thiago C. Santini, Thomas Kübler, and Enkelejda Kasneci. 2016. ElSe: Ellipse selection for robust pupil detection in real-world environments. In Eye Tracking Research and Applications Symposium (ETRA), Vol. 14. 123--130. https://doi.org/10.1145/2857491.2857505
[14]
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International conference on machine learning. PMLR, 1180--1189.
[15]
Stephan J. Garbin, Yiru Shen, Immo Schuetz, Robert Cavin, Gregory Hughes, and Sachin S. Talathi. 2019. OpenEDS: Open Eye Dataset. (2019). http://arxiv.org/abs/1905.03702
[16]
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. 2015. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision. 2551--2559.
[17]
Agostino Gibaldi, Vasha DuTell, and Martin S Banks. 2021. Solving Parallax Error for 3D Eye Tracking. In ACM Symposium on Eye Tracking Research and Applications. 1--4.
[18]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014).
[19]
Ishaan Gulrajani and David Lopez-Paz. 2020. In Search of Lost Domain Generalization. (2020). arXiv:2007.01434 http://arxiv.org/abs/2007.01434
[20]
Alex Hernández-García and Peter König. 2018. Further advantages of data augmentation on convolutional neural networks. In International Conference on Arti!cial Neural Networks. Springer, 95--103.
[21]
Bincheng Huang, Si Chen, Fan Zhou, Cheng Zhang, and Feng Zhang. 2020. Episodic Training for Domain Generalization Using Latent Domains. In International Conference on Cognitive Systems and Signal Processing. Springer, 85--93.
[22]
Sergey Io#e and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. PMLR, 448--456.
[23]
Paul Jaccard. 1912. The distribution of the !ora in the alpine zone. 1. New phytologist 11, 2 (1912), 37--50.
[24]
Simon Jegou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. 2017. The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. IEEE Computer Society Conference on Computer Proc. ACM Hum.-Comput. Interact., Vol. 6, No. ETRA, Article 139. Publication date: May 2022. 139:16 Rakshit S. Kothari et al. Vision and Pattern Recognition Workshops 2017-July (2017), 1175--1183. https://doi.org/10.1109/CVPRW.2017.156
[25]
Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, Christoph Reinders, Sarthak Yadav, Joy Banerjee, Gábor Vecsei, Adam Kraft, Zheng Rui, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian Pfei#er, Ben Cook, Ismael Fernández, François-Michel De Rainville, Chi-Hung Weng, Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et al. 2020. imgaug. https://github.com/aleju/imgaug. Online; accessed 01-Feb-2020.
[26]
Priya Kansal and Sabarinathan Devanathan. 2019. Eyenet: Attention based convolutional encoder-decoder network for eye region segmentation. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 3688--3693.
[27]
Enkelejda Kasneci, Katrin Sippel, Kathrin Aehling, Martin Heister, Wolfgang Rosenstiel, Ulrich Schiefer, and Elena Papageorgiou. 2014. Driving with binocular visual "eld loss? A study on a supervised on-road parcours with simultaneous eye and head tracking. PloS one 9, 2 (2014), e87470.
[28]
Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction. UbiComp 2014 - Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2014), 1151--1160. https://doi.org/10.1145/2638728.2641695
[29]
Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems 30 (2017).
[30]
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A. Efros, and Antonio Torralba. 2012. Undoing the damage of dataset bias. Lecture Notes in Computer Science (including subseries Lecture Notes in Arti!cial Intelligence and Lecture Notes in Bioinformatics) 7572 LNCS, PART 1 (2012), 158--171. https://doi.org/10.1007/978--3--642--33718--5_12
[31]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. AIP Conference Proceedings 1631, 2 (12 2014), 58--62. https://doi.org/10.1016/j.jneumeth.2005.04.009
[32]
Rakshit S Kothari, Aayush K Chaudhary, Reynold J Bailey, Je# B Pelz, and Gabriel J Diaz. 2021. EllSeg: An Ellipse Segmentation Framework for Robust Gaze Tracking. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2757--2767.
[33]
Wouter M Kouw and Marco Loog. 2018. An introduction to domain adaptation and transfer learning. arXiv preprint arXiv:1812.11806 (2018).
[34]
Wouter Marco Kouw and Marco Loog. 2019. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence (2019).
[35]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. 2018. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI Conference on Arti!cial Intelligence, Vol. 32.
[36]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2017. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision. 5542--5550.
[37]
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. 2018. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5400--5409.
[38]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).
[39]
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. 2013. Domain generalization via invariant feature representation. 30th International Conference on Machine Learning, ICML 2013 PART 1 (2013), 10--18. arXiv:1301.2115
[40]
Nitinraj Nair, Rakshit Kothari, Aayush K Chaudhary, Zhizhuo Yang, Gabriel J Diaz, Je# B Pelz, and Reynold J Bailey. 2020. RIT-Eyes: Rendering of near-eye images for eye-tracking applications. In ACM Symposium on Applied Perception 2020. 1--9.
[41]
Jakub Nalepa, Michal Marcinkiewicz, and Michal Kawulok. 2019. Data augmentation for brain-tumor segmentation: a review. Frontiers in computational neuroscience 13 (2019), 83.
[42]
Seonwook Park, Adrian Spurr, and Otmar Hilliges. 2018. Deep Pictorial Gaze Estimation. Vol. 11217 LNCS. 741--757. https://doi.org/10.1007/978--3-030-01261--8{_}44
[43]
Seonwook Park, Xucong Zhang, Andreas Bulling, and Otmar Hilliges. 2018. Learning to "nd eye region landmarks for remote gaze estimation in unconstrained settings. In Eye Tracking Research and Applications Symposium (ETRA). ACM Press, New York, New York, USA, 1--10. https://doi.org/10.1145/3204493.3204545
[44]
Jean Ponce, Tamara L Berg, Mark Everingham, David A Forsyth, Martial Hebert, Svetlana Lazebnik, Marcin Marszalek, Cordelia Schmid, Bryan C Russell, Antonio Torralba, et al. 2006. Dataset issues in object recognition. In Toward category-level object recognition. Springer, 29--48.
[45]
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. 2020. Understanding and mitigating the tradeo# between robustness and accuracy. arXiv preprint arXiv:2002.10716 (2020).
[46]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. 9351 (2015), 234--241. https://doi.org/10.1007/978--3--319--24574--4{_}28
[47]
Thiago Santini, Diederick C. Niehorster, and Enkelejda Kasneci. 2019. Get a grip: Slippage-robust and glint-free gaze estimation for real-time pervasive head-mounted eye tracking. Eye Track. Res. Appl. Symp. (2019). https: Proc. ACM Hum.-Comput. Interact., Vol. 6, No. ETRA, Article 139. Publication date: May 2022. EllSeg-Gen, towards Domain Generalization for head-mounted eyetracking 139:17 //doi.org/10.1145/3314111.3319835
[48]
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. 2018. How does batch normalization help optimization?. In Proceedings of the 32nd international conference on neural information processing systems. 2488--2498.
[49]
Yiru Shen, Oleg Komogortsev, and Sachin S Talathi. 2020. Domain Adaptation for Eye Segmentation. In European Conference on Computer Vision. Springer, 555--569.
[50]
Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data 6, 1 (2019), 1--48.
[51]
TT Tanimoto. 1968. An elementary mathematical theory of classi"cation and prediction, IBM Report (November, 1958), cited in: G. Salton, Automatic Information Organization and Retrieval.
[52]
Martin A Tanner and Wing Hung Wong. 1987. The calculation of posterior distributions by data augmentation. Journal of the American statistical Association 82, 398 (1987), 528--540.
[53]
Marc Tonsen, Xucong Zhang, Yusuke Sugano, and Andreas Bulling. 2016. Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments. In Proceedings of the ninth biennial ACM symposium on eye tracking research & applications. 139--142.
[54]
Antonio Torralba and Alexei A Efros. 2011. Unbiased look at dataset bias. In CVPR 2011. IEEE, 1521--1528.
[55]
Eric Tzeng, Judy Ho#man, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7167--7176.
[56]
Eric Tzeng, Judy Ho#man, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014).
[57]
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016).
[58]
David A Van Dyk and Xiao-Li Meng. 2001. The art of data augmentation. Journal of Computational and Graphical Statistics 10, 1 (2001), 1--50.
[59]
Vladimir N Vapnik. 1999. An overview of statistical learning theory. IEEE transactions on neural networks 10, 5 (1999), 988--999.
[60]
F. J. Vera-Olmos, E. Pardo, H. Melero, and N. Malpica. 2018. DeepEye: Deep convolutional network for pupil detection in real environments. Integrated Computer-Aided Engineering 26, 1 (2018), 85--95. https://doi.org/10.3233/ICA-180584
[61]
Jason Wang, Luis Perez, et al. 2017. The e#ectiveness of data augmentation in image classi"cation using deep learning. Convolutional Neural Networks Vis. Recognit 11 (2017), 1--8.
[62]
Erroll Wood and Andreas Bulling. 2014. Eyetab: Model-based gaze estimation on unmodi"ed tablet computers. In Proceedings of the symposium on eye tracking research and applications. 207--210.
[63]
Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. 2020. A closer look at accuracy vs. robustness. Advances in neural information processing systems 33 (2020), 8588--8601.
[64]
Yuk-Hoi Yiu, Moustafa Aboulatta, Theresa Raiser, Leoni Ophey, Virginia L. Flanagin, Peter zu Eulenburg, and SeyedAhmad Ahmadi. 2019. DeepVOG: Open-source pupil segmentation and gaze estimation in neuroscience using deep learning. Journal of Neuroscience Methods 324 (2019), 108307. https://doi.org/10.1016/j.jneumeth.2019.05.016
[65]
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-o# between robustness and accuracy. In International Conference on Machine Learning. PMLR, 7472-- 7482.

Cited By

View all
  • (2024)Improving Eye-Tracking Data Quality: A Framework for Reproducible Evaluation of Detection AlgorithmsSensors10.3390/s2409268824:9(2688)Online publication date: 24-Apr-2024
  • (2024)Using Deep Learning to Increase Eye-Tracking Robustness, Accuracy, and Precision in Virtual RealityProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547057:2(1-16)Online publication date: 17-May-2024
  • (2024)Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)Proceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547047:2(1-16)Online publication date: 17-May-2024
  • Show More Cited By

Index Terms

  1. EllSeg-Gen, towards Domain Generalization for Head-Mounted Eyetracking

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Human-Computer Interaction
      Proceedings of the ACM on Human-Computer Interaction  Volume 6, Issue ETRA
      ETRA
      May 2022
      198 pages
      EISSN:2573-0142
      DOI:10.1145/3537904
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 13 May 2022
      Published in PACMHCI Volume 6, Issue ETRA

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. domain generalization
      2. semantic segmentation

      Qualifiers

      • Research-article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)67
      • Downloads (Last 6 weeks)8
      Reflects downloads up to 30 Aug 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Improving Eye-Tracking Data Quality: A Framework for Reproducible Evaluation of Detection AlgorithmsSensors10.3390/s2409268824:9(2688)Online publication date: 24-Apr-2024
      • (2024)Using Deep Learning to Increase Eye-Tracking Robustness, Accuracy, and Precision in Virtual RealityProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547057:2(1-16)Online publication date: 17-May-2024
      • (2024)Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)Proceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547047:2(1-16)Online publication date: 17-May-2024
      • (2024)Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking SystemsProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547037:2(1-17)Online publication date: 17-May-2024
      • (2024)GEARS: Generalizable Multi-Purpose Embeddings for Gaze and Hand Data in VR InteractionsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659551(279-289)Online publication date: 22-Jun-2024
      • (2024)Eye-tracking on virtual reality: a surveyVirtual Reality10.1007/s10055-023-00903-y28:1Online publication date: 5-Feb-2024
      • (2023)A framework for generalizable neural networks for robust estimation of eyelids and pupilsBehavior Research Methods10.3758/s13428-023-02266-356:4(3959-3981)Online publication date: 28-Nov-2023

      View Options

      Get Access

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media