Abstract
Recently, deep learning approaches have been proposed to detect eye movements such as fixations, saccades, and smooth pursuits from eye tracking data. These are end-to-end methods that have shown to surpass traditional ones, requiring no ad hoc parameters. In this work we propose the use of temporal convolutional networks (TCNs) for automated eye movement classification and investigate the influence of feature space, scale, and context window sizes on the classification results. We evaluated the performance of TCNs against a state-of-the-art 1D-CNN-BLSTM model using GazeCom, a public available dataset. Our results show that TCNs can outperform the 1D-CNN-BLSTM, achieving an F-score of 94.2% for fixations, 89.9% for saccades, and 73.7% for smooth pursuits on sample level, and 89.6%, 94.3%, and 60.2% on event level. We also state the advantages of TCNs over sequential networks for this problem, and how these scores can be further improved by feature space extension.
Supported by the São Paulo Research Foundation, grant 2015/26802-1.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agtzidis, I., Startsev, M., Dorr, M.: In the pursuit of (ground) truth: a hand-labelling tool for eye movements recorded during dynamic scene viewing. In: 2016 IEEE Second Workshop on Eye Tracking and Visualization (ETVIS), pp. 65–68 (2016)
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. CoRR abs/1803.01271 (2018). http://arxiv.org/abs/1803.01271
Berg, D.J., Boehnke, S.E., Marino, R.A., Munoz, D.P., Itti, L.: Free viewing of dynamic stimuli by humans and monkeys. J. Vis. 9(5), 19 (2009). https://doi.org/10.1167/9.5.19
Berndt, S., Kirkpatrick, D., Taviano, T., Komogortsev, O.: Tertiary eye movement classification by a hybrid algorithm. CoRR abs/1904.10085 (2019). http://arxiv.org/abs/1904.10085
Campbell, C.S., Maglio, P.P.: A robust algorithm for reading detection. In: Proceedings of the 2001 workshop on Perceptive User Interfaces, pp. 1–7 (2001)
Cassin, B., Rubin, M.L., Solomon, S.: Dictionary of Eye Terminology, vol. 10. Triad Publishing Company, Gainsville (1984)
Dorr, M., Martinetz, T., Gegenfurtner, K.R., Barth, E.: Variability of eye movements when viewing dynamic natural scenes. J. Vision 10(10), 28 (2010). https://doi.org/10.1167/10.10.28
Duchowski, A.T.: Gaze-based interaction: a 30 year retrospective. Comput. Graph. 73, 59–69 (2018). https://doi.org/10.1016/j.cag.2018.04.002
Fuhl, W.: Fully convolutional neural networks for raw eye tracking data segmentation, generation, and reconstruction (2020)
Goodfellow, I.J., Bengio, Y., Courville, A.C.: Deep Learning: Adaptive computation and Machine Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org/
Hooge, I., Niehorster, D., Nyström, M., Andersson, R., Hessels, R.: Is human classification by experienced untrained observers a gold standard in fixation detection? Behav. Res. Methods 50(5), 1864–1881 (2018). https://doi.org/10.3758/s13428-017-0955-x
Hoppe, S., Bulling, A.: End-to-end eye movement detection using convolutional neural networks. CoRR abs/1609.02452 (2016). http://arxiv.org/abs/1609.02452
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980
Komogortsev, O., Gobert, D., Jayarathna, S., Koh, D., Gowda, S.: Standardization of automated analyses of oculomotor fixation and saccadic behaviors. IEEE Trans. Biomed. Eng. 57, 2635–2645 (2010). https://doi.org/10.1109/TBME.2010.2057429
Komogortsev, O., Karpov, A.: Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades. Behav. Res. Methods 45 (2012). https://doi.org/10.3758/s13428-012-0234-9
Komogortsev, O.V., Karpov, A.: Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades. Behav. Res. Methods 45(1), 203–215 (2013). https://doi.org/10.3758/s13428-012-0234-9
Komogortsev, O.V., Khan, J.I.: Kalman filtering in the design of eye-gaze-guided computer interfaces. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4552, pp. 679–689. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73110-8_74. http://dl.acm.org/citation.cfm?id=1769590.1769667
Larsson, L., Nyström, M., Andersson, R., Stridh, M.: Detection of fixations and smooth pursuit movements in high-speed eye-tracking data. Biomed. Signal Process. Control 18, 145–152 (2015). https://doi.org/10.1016/j.bspc.2014.12.008, http://www.sciencedirect.com/science/article/pii/S1746809414002031
Leigh, R.J., Zee, D.S.: The neurology of eye movements. OUP USA (2015)
Nyström, M., Holmqvist, K.: An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behav. Res. Methods 42(1), 188–204 (2010). https://doi.org/10.3758/BRM.42.1.188
Peters, C., Pelachaud, C., Bevacqua, E., Mancini, M., Poggi, I.: A model of attention and interest using gaze behavior. In: Panayiotopoulos, T., Gratch, J., Aylett, R., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 229–240. Springer, Heidelberg (2005). https://doi.org/10.1007/11550617_20
Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, ETRA 2000, pp. 71–78. ACM, New York (2000). https://doi.org/10.1145/355017.355028, http://doi.acm.org/10.1145/355017.355028
Santini, T., Fuhl, W., Kübler, T., Kasneci, E.: Bayesian identification of fixations, saccades, and smooth pursuits. In: Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, ETRA 2016, pp. 163–170. ACM, New York (2016). https://doi.org/10.1145/2857491.2857512, http://doi.acm.org/10.1145/2857491.2857512
Sauter, D., Martin, B.J., Di Renzo, N., Vomscheid, C.: Analysis of eye tracking movements using innovations generated by a kalman filter. Med. Biol. Eng. Comput. 29(1), 63–69 (1991). https://doi.org/10.1007/BF02446297
Shepherd, S.: Following gaze: Gaze-following behavior as a window into social cognition. Front. Integr. Neurosci. 4, 5 (2010). https://doi.org/10.3389/fnint.2010.00005, https://www.frontiersin.org/articles/10.3389/fnint.2010.00005
Startsev, M., Agtzidis, I., Dorr, M.: 1D CNN with BLSTM for automated classification of fixations, saccades, and smooth pursuits. Behav. Res. Methods 51(2), 556–572 (2019). https://doi.org/10.3758/s13428-018-1144-2
Startsev, M., Agtzidis, I., Dorr, M.: Sequence-to-sequence deep learning for eye movement classification. In: Perception, vol. 48, pp. 200–200. Sage Publications LTD., London (2019)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, Quebec, Canada, 8–13 December 2014, pp. 3104–3112 (2014). http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks
Vidal, M., Bulling, A., Gellersen, H.: Detection of smooth pursuits using eye movement shape features. In: Proceedings of the Symposium on Eye Tracking Research and Applications, ETRA 2012, pp. 177–180. ACM, New York (2012). https://doi.org/10.1145/2168556.2168586, http://doi.acm.org/10.1145/2168556.2168586
Zemblys, R., Niehorster, D.C., Holmqvist, K.: gazenet: End-to-end eye-movement event detection with deep neural networks. Behav. Res. Methods 51, 840–864 (2018)
Zemblys, R., Niehorster, D.C., Komogortsev, O., Holmqvist, K.: Using machine learning to detect events in eye-tracking data. Behav. Res. Methods 50(1), 160–181 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Elmadjian, C., Gonzales, C., Morimoto, C.H. (2021). Eye Movement Classification with Temporal Convolutional Networks. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12663. Springer, Cham. https://doi.org/10.1007/978-3-030-68796-0_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-68796-0_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68795-3
Online ISBN: 978-3-030-68796-0
eBook Packages: Computer ScienceComputer Science (R0)
