Residual feature learning with hierarchical calibration for gaze estimation
Abstract
Gaze estimation aims to predict accurate gaze direction from natural eye images, which is an extreme challenging task due to both random variations in head pose and person-specific biases. Existing works often independently learn features from binocular images and directly concatenate them for gaze estimation. In this paper, we propose a simple yet effective two-stage framework for gaze estimation, in which both residual feature learning (RFL) and hierarchical gaze calibration (HGC) networks are designed to consistently improve the performance of gaze estimation. Specifically, the RFL network extracts informative features by jointly exploring the symmetric and asymmetric factors between left and right eyes, which can produce accurate initial predictions as much as possible. Besides, the HGC network cascades a personal-specific transform module to further transform the distribution of gaze point from coarse to fine, which can effectively compensate the subjective bias in initial predictions. Extensive experiments on both EVE and MPIIGaze datasets show that our method outperforms the state-of-the-art approaches.
References
[1]
Konrad R, Angelopoulos A, and Wetzstein G Gaze-contingent ocular parallax rendering for virtual reality ACM Trans. Graph. (TOG) 2020 39 2 1-12
[2]
Sitzmann V, Serrano A, Pavel A, Agrawala M, Gutierrez D, Masia B, and Wetzstein G Saliency in vr: how do people explore virtual environments? IEEE Trans. Visual Comput. Graph. 2018 24 4 1633-1642
[3]
Zhang, X., Sugano, Y., Bulling, A.: Everyday eye contact detection using unsupervised gaze target discovery. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 193–203 (2017)
[4]
Terzioğlu, Y., Mutlu, B., Şahin, E.: Designing social cues for collaborative robots: the role of gaze and breathing in human–robot collaboration. In: Proceedings of the 2020 ACM/IEEE International Conference on Human–Robot Interaction, pp. 343–357 (2020)
[5]
Gerber, M.A., Schroeter, R., Xiaomeng, L., Elhenawy, M.: Self-interruptions of non-driving related tasks in automated vehicles: mobile vs head-up display. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–9 (2020)
[6]
Palazzi A, Abati D, Solera F, Cucchiara R, et al. Predicting the driver’s focus of attention: the dr (eye) ve project IEEE Trans. Pattern Anal. Mach. Intell. 2018 41 7 1720-1733
[7]
Jin, S., Dai, J., Nguyen, T.: Kappa angle regression with ocular counter-rolling awareness for gaze estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2658–2667 (2023)
[8]
Li, Y., Zhan, Y., Yang, Z.: Evaluation of appearance-based eye tracking calibration data selection. In: 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), pp. 222–224. IEEE (2020)
[9]
Park, S., Aksan, E., Zhang, X., Hilliges, O.: Towards end-to-end video-based eye-tracking. In: Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pp. 747–763. Springer (2020)
[10]
Bao J, Liu B, and Yu J An individual-difference-aware model for cross-person gaze estimation IEEE Trans. Image Process. 2022 31 3322-3333
[11]
Miao, Q., Hoai, M., Samaras, D.: Patch-level gaze distribution prediction for gaze following. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 880–889 (2023)
[12]
Ghosh, S., Dhall, A., Hayat, M., Knibbe, J., Ji, Q.: Automatic gaze analysis: a survey of deep learning based approaches. arXiv preprint arXiv:2108.05479 (2021)
[13]
Zhang X, Sugano Y, Fritz M, and Bulling A Mpiigaze: real-world dataset and deep appearance-based gaze estimation IEEE Trans. Pattern Anal. Mach. Intell. 2017 41 1 162-175
[14]
Model, D., Eizenman, M.: User-calibration-free remote eye-gaze tracking system with extended tracking range. In: 2011 24th Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 001268–001271. IEEE (2011)
[15]
Wang, K., Ji, Q.: Real time eye gaze tracking with kinect. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2752–2757. IEEE (2016)
[16]
Liu J, Chi J, Hu W, and Wang Z 3d model-based gaze tracking via iris features with a single camera and a single light source IEEE Trans. Hum. Mach. Syst. 2020 51 2 75-86
[17]
Chen, J., Ji, Q.: 3d gaze estimation with a single camera without ir illumination. In: 2008 19th International Conference on Pattern Recognition, pp. 1–4. IEEE (2008)
[18]
Wu, Y., Ji, Q.: Learning the deep features for eye detection in uncontrolled conditions. In: 2014 22nd International Conference on Pattern Recognition, pp. 455–459. IEEE (2014)
[19]
Wood, E., Baltrušaitis, T., Morency, L.-P., Robinson, P., Bulling, A.: A 3d morphable eye region model for gaze estimation. In: Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pp. 297–313. Springer (2016)
[20]
Fischer, T., Chang, H.J., Demiris, Y.: Rt-gene: real-time eye gaze estimation in natural environments. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 334–352 (2018)
[21]
Cheng, Y., Lu, F., Zhang, X.: Appearance-based gaze estimation via evaluation-guided asymmetric regression. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 100–115 (2018)
[22]
Cheng, Y., Huang, S., Wang, F., Qian, C., Lu, F.: A coarse-to-fine adaptive network for appearance-based gaze estimation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 10623–10630 (2020)
[23]
Bao, Y., Cheng, Y., Liu, Y., Lu, F.: Adaptive feature fusion network for gaze tracking in mobile tablets. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9936–9943. IEEE (2021)
[24]
Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: It’s written all over your face: full-face appearance-based gaze estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 51–60 (2017)
[25]
Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., Torralba, A.: Eye tracking for everyone. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2176–2184 (2016)
[26]
Cheng, Y., Lu, F.: Gaze estimation using transformer. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 3341–3347. IEEE (2022)
[27]
Cheng, Y., Bao, Y., Lu, F.: Puregaze: purifying gaze feature for generalizable gaze estimation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 436–443 (2022)
[28]
Park, S., Mello, S.D., Molchanov, P., Iqbal, U., Hilliges, O., Kautz, J.: Few-shot adaptive gaze estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9368–9377 (2019)
[29]
Xiong, Y., Kim, H.J., Singh, V.: Mixed effects neural networks (menets) with applications to gaze estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7743–7752 (2019)
[30]
Lindén, E., Sjostrand, J., Proutiere, A.: Learning to personalize in appearance-based gaze tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0 (2019)
[31]
He, J., Pham, K., Valliappan, N., Xu, P., Roberts, C., Lagun, D., Navalpakkam, V.: On-device few-shot personalization for real-time gaze estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0 (2019)
[32]
Guo, Z., Yuan, Z., Zhang, C., Chi, W., Ling, Y., Zhang, S.: Domain adaptation gaze estimation by embedding with prediction consistency. In: Proceedings of the Asian Conference on Computer Vision (2020)
[33]
Wang W, Shen J, Dong X, Borji A, and Yang R Inferring salient objects from human fixations IEEE Trans. Pattern Anal. Mach. Intell. 2019 42 8 1913-1927
[34]
Kruthiventi, S.S., Gudisa, V., Dholakiya, J.H., Babu, R.V.: Saliency unified: a deep architecture for simultaneous eye fixation prediction and salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5781–5790 (2016)
[35]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)
[36]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
[37]
Zhang, X., Park, S., Beeler, T., Bradley, D., Tang, S., Hilliges, O.: Eth-xgaze: A large scale dataset for gaze estimation under extreme head pose and gaze variation. In: Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pp. 365–381. Springer (2020)
[38]
Cheng, Y., Zhang, X., Lu, F., Sato, Y.: Gaze estimation by exploring two-eye asymmetry. IEEE Trans. Image Process. 29, 5259–5272 (2020)
[39]
Biswas, P., et al.: Appearance-based gaze estimation using attention and difference mechanism. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3143–3152 (2021)
[40]
Abdelrahman, A.A., Hempel, T., Khalifa, A., Al-Hamadi, A.: L2cs-net: fine-grained gaze estimation in unconstrained environments. arXiv preprint arXiv:2203.03339 (2022)
Index Terms
- Residual feature learning with hierarchical calibration for gaze estimation
Index terms have been assigned to the content through auto-classification.
Recommendations
Eye-Model-Based Gaze Estimation by RGB-D Camera
CVPRW '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition WorkshopsThis paper proposes a method of eye-model-based gaze estimation by RGB-D camera, Kinect sensor. Different from other methods, our method sets up a model to calibrate the eyeball center by gazing at a target in 3D space, not predefined. And then by ...
Eye-gaze estimation under various head positions and iris states
A new method for eye-gaze estimation under normal head movement is investigated.Head position and orientations are acquired by Kinect depth data and eye direction is obtained from high resolution images.Bayesian multinomial logistic regression is used ...
Gaze from Head: Gaze Estimation Without Observing Eye
Pattern RecognitionAbstractWe propose a gaze estimation method not from eye observation but from head motion. This proposed method is based on physiological studies about the eye-head coordination, and the gaze direction is estimated from observation of head motion by using ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 05 May 2024
Accepted: 17 April 2024
Revision received: 16 April 2024
Received: 26 December 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Natural Science Foundation of Shaanxi Province
- China Postdoctoral Science Foundation
- Fundamental Research Funds for the Central Unversities
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 08 Feb 2025