On Eye Tracking in Software Engineering
Abstract
Eye tracking is becoming more and more important as a research method within the field of software engineering (SE). Existing meta-analyses focus on the design or conduct of SE eye tracking studies rather than the analysis phase. This article attempts to fill this gap; it presents a systematic literature review of eye tracking studies in the field of SE—focusing mainly on the data analysis methods used. From the IEEE Xplore and ACM digital libraries we gather 125 papers up to the first quarter of 2024. Detailed evaluation provides information on the number of papers that use specific methods of analysis (i.e., descriptive or inferential statistics, and gaze visualization) or settings (e.g., sample size, technical setup, and selected aspects of research design). With the data obtained we can infer the popularity of specific analysis methods in the field. Those results enable efficient work on data analysis tools or education of aspiring researchers and can serve as basis for standardization or guidelines within the community—providing for methods to include as well as current inconsistencies.
References
[1]
Abid N, Maletic J, Sharif B. Using developer eye movements to externalize the mental model used in code summarization tasks. ACM; 2019. pp. 1–9.
[2]
Abid N, Sharif B, Dragan N, et al. Developer reading behavior while summarizing java methods: size and context matters, vol. 2019-May. IEEE; 2019. pp. 384–395,.
[3]
Ahmad H, Karas Z, Diaz K, et al. How do we read formal claims? eye-tracking and the cognition of proofs about algorithms. IEEE; 2023. pp. 208–220.
[4]
Ahrens M, Schneider K, Busch M. Attention in software maintenance: an eye tracking study. IEEE; 2019. pp. 2–9.
[5]
Ahsan Z, Obaidellah U. Predicting expertise among novice programmers with prior knowledge on programming tasks. IEEE; 2020. pp. 1008–1016.
[6]
Ahsan Z, Obaidellah U. Is clustering novice programmers possible? Investigating scanpath trend analysis in programming tasks. ACM; 2023. pp. 1–7.
[7]
Alcocer J, Cossio-Chavalier A, Rojas-Stambuk T, et al. An eye-tracking study on the use of split/unified code change views for bug detection IEEE Access 2023 11 136195-136205
[8]
Ali N, Sharafl Z, Guéhéneuc YG, et al. An empirical study on requirements traceability using eye-tracking. IEEE; 2012. pp. 191–200.
[9]
Aljehane S, Sharif B, Maletic JI. Determining differences in reading behavior between experts and novices by investigating eye movement on source code constructs during a bug fixing task, vol. PartF169257. ACM; 2021. pp. 1–6.
[10]
Aljehane S, Sharif B, Maletic J. Studying developer eye movements to measure cognitive workload and visual effort for expertise assessment, vol. 7. In: PACM HCI. 2023. pp. 1–18.
[11]
Al Madi N, Busjahn T, and Sharif B Summary of the tenth international workshop on eye movements in programming (EMIP 2022) ACM SIGSOFT SEN 2023 48 79-80
[12]
Andrzejewska M and Stolińska A Do structured flowcharts outperform pseudocode? Evidence from eye movements IEEE Access 2022 10 132965-132975
[13]
Balzert H Lehrbuch der Softwaretechnik: Basiskonzepte und Requirements Engineering 2009 3 Heidelberg Spektrum Akademischer Verlag
[14]
Barik T, Smith J, Lubick K, et al. Do developers read compiler error messages? IEEE; 2017. pp. 575–585.
[15]
Bauer J, Siegmund J, Peitek N, et al. Indentation: simply a matter of style or support for program comprehension? vol 2019-May. IEEE; 2019. pp. 154–164.
[16]
Bednarik R, Tukiainen M. Effects of display blurring on the behavior of novices and experts during program debugging. ACM; 2005. pp. 1204–1207.
[17]
Bednarik R, Tukiainen M. An eye-tracking methodology for characterizing program comprehension processes. ACM; 2006. pp. 125–132.
[18]
Bednarik R, Tukiainen M. Temporal eye-tracking data: evolution of debugging strategies with multiple representations. ACM; 2008. pp. 99–102;.
[19]
Bednarik R, Schulte C, Budde L, et al. Eye-movement modeling examples in source code comprehension: a classroom study. ACM; 2018. pp. 1–8.
[20]
Beelders T. Eye-tracking analysis of source code reading on a line-by-line basis. ACM; 2022. pp. 1–7.
[21]
Beelders T, Plessis JPD. The influence of syntax highlighting on scanning and reading behaviour for source code, vol. 26–28-September-2016. ACM; 2016. pp. 1–10.
[22]
Begel A, Vrzakova H. Eye movements in code review. ACM; 2018. pp. 1–5.
[23]
Bernard L, Raina S, Taylor B, et al. Minimizing cognitive load in cyber learning materials—an eye tracking study, vol. PartF169257. ACM; 2021. pp. 1–6,.
[24]
Bertram I, Hong J, Huang Y, et al. Trustworthiness perceptions in code review: an eye-tracking study. IEEE Computer Society; 2020. pp. 1–6.
[25]
Blascheck T, Sharif B. Visually analyzing eye movements on natural language texts and source code snippets. ACM; 2019. pp. 1–9.
[26]
Busjahn T, Tamm S. A deeper analysis of AOI coverage in code reading. ACM; 2021. pp. 1–7.
[27]
Busjahn T, Schulte C, Busjahn A. Analysis of code reading to gain more insight in program comprehension. ACM; 2011. pp. 1–9.
[28]
Busjahn T, Bednarik R, Schulte C. What influences dwell time during source code reading? Analysis of element type and frequency as factors. ACM; 2014. pp. 335–338.
[29]
Busjahn T, Bednarik R, Begel A, et al. Eye movements in code reading: relaxing the linear order, vol. 2015-August. IEEE; 2015. pp. 255–265.
[30]
Busjahn T, Simon, Paterson JH. Looking at the main method—an educator’s perspective. ACM; 2021. pp. 1–10.
[31]
Chandrika K, Amudha J, Sudarsan SD. Recognizing eye tracking traits for source code review. IEEE; 2017. pp. 1–8.
[32]
Cheng G, Poon L, Lau W, et al. Applying eye tracking to identify students’ use of learning strategies in understanding program code. ACM; 2019. pp. 140–144.
[33]
Chitalkina N, Bednarik R, Puurtinen M, et al. When you ignore what you see: How to study proof-readers’ error in pseudocode reading. ACM; 2020. pp. 1–5.
[34]
Costa JSD, Gheyi R. Evaluating the code comprehension of novices with eye tracking. 2023. pp. 332–341.
[35]
Crosby M and Stelovsky J How do we read algorithms? A case study Computer 1990 23 24-35
[36]
D’Angelo S, Begel A. Improving communication between pair programmers using shared gaze awareness, vol. 2017-January. ACM; 2017. pp. 6245–6255.
[37]
Davis D, Zhu F. Understanding and improving secure coding behavior with eye tracking methodologies. ACM; 2020. pp. 107–114.
[38]
Fakhoury S, Ma Y, Arnaoudova V, et al. The effect of poor source code lexicon and readability on developers’ cognitive load. ACM; 2018. pp. 286–296.
[39]
Gorski P, Möller S, Wiefling S, et al. ‘I just looked for the solution!’ on integrating security-relevant information in non-security API documentation to support secure coding practices IEEE Trans Softw Eng 2022 48 3467-3484
[40]
Goswami A, Walia G, McCourt M, et al. Using eye tracking to investigate reading patterns and learning styles of software requirement inspectors to enhance inspection team outcome. ACM; 2016. pp. 1–10.
[41]
Grabinger L, Hauser F, Mottok J. On the perception of graph layouts. J Softw (Malden) Early View 2023;1–18.
[42]
Gralha C, ao MG, ao Araújo J. Analysing gender differences in building social goal models: a quasi-experiment, vol. 2019-September. IEEE; 2019. pp. 165–176.
[43]
Gralha C, Pereira R, Goulão M, et al. On the impact of using different templates on creating and understanding user stories. IEEE; 2021. pp. 209–220.
[44]
Gunawan F, Wijaya O, Soewito B, et al. An analysis of concentration region on powerpoint slides using eye tracking. IEEE; 2017. pp. 1–5.
[45]
Han L, Chen T, Demartini G, et al. A data-driven analysis of behaviors in data curation processes ACM Trans Inf Syst 2023 41 1-35
[46]
Hauser F, Mottok J, Gruber H. Eye tracking metrics in software engineering. ACM; 2018. pp. 39–44.
[47]
Hauser F, Schreistetter S, Reuter R, et al. Code reviews in C++: preliminary results from an eye tracking study. ACM; 2020. pp. 1–5.
[48]
Hauser F, Grabinger L, Mottok J, et al. Visual expertise in code reviews: using holistic models of image perception to analyze and interpret eye movements. ACM; 2023. pp. 1–7.
[49]
Hejmady P, Narayanan H. Multiple visualizations and debugging: how do we co-ordinate these? ACM; 2012. pp. 1547–1552.
[50]
Hejmady P, Narayanan H. Visual attention patterns during program debugging with an IDE. ACM; 2012. pp. 197–200.
[51]
Ho HF, Huang DH. Differences of reading processes between and gate and or gate: evidences from eye movement. IEEE; 2014. pp. 123–128.
[52]
Holmqvist K, Nyström M, Andersson R, et al. Eye tracking: a comprehensive guide to methods and measures 2011 Oxford Oxford University Press
[53]
Homann A, Grabinger L, Hauser F, et al. An eye tracking study on MISRA C coding guidelines. ACM; 2023. pp. 130–137.
[54]
Hüsing S, Schulte C, Sparmann S, et al. Using worked examples for engaging in epistemic programming projects. 2024. pp. 443–449.
[55]
Iwasa H, Akashi T, Ohno S. A fundamental study on gaze tracking for programming learners based on hierarchical segmentation. 2023. pp. 558–559.
[56]
Jbara A, Feitelson D. How programmers read regular code: A controlled experiment using eye tracking, vol. 2015-August. IEEE; 2015. pp. 244–254.
[57]
Jermann P, Nüssli MA. Effects of sharing text selections on gaze cross-recurrence and interaction quality in a pair programming task. ACM; 2012. pp. 1125–1134.
[58]
Jermann P, Sharma K. Gaze as a proxy for cognition and communication. IEEE; 2018. pp. 152–154.
[59]
Kano T, Sakagami R, Akakura T. Modeling of cognitive processes based on gaze transition during programming debugging. IEEE; 2021. pp. 412–413.
[60]
Karras O, Risch A, Schneider K. Interrelating use cases and associated requirements by links: an eye tracking study on the impact of different linking variants on the reading behavior, vol. Part F137700. ACM; 2018. pp. 2–12.
[61]
Kather P, Duran R, and Vahrenhold J Through (tracking) their eyes: abstraction and complexity in program comprehension ACM Trans Comput 2022 22 1-33
[62]
Katona J, Kovari A, Costescu C, et al. The examination task of source-code debugging using GP3 eye tracker. IEEE; 2019. pp. 329–333.
[63]
Katona J, Kovari A, Heldal I, et al. Using eye-tracking to examine query syntax and method syntax comprehension in LINQ. IEEE; 2020. pp. 437–444.
[64]
Kevic K. Using eye gaze data to recognize task-relevant source code better and more fine-grained. IEEE; 2017. pp. 103–105.
[65]
Kevic K, Walters B, Shaffer T, et al. Tracing software developers’ eyes and interactions for change tasks. ACM; 2015. pp. 202–213.
[66]
Kilic O, Say B, Demirörs O. Cognitive aspects of error finding on a simulation conceptual modeling notation. IEEE; 2008. pp. 1–6.
[67]
Kitchenham B. Guidelines for performing systematic literature reviews in software engineering. 2007.
[68]
Konopka M, Talian A, Tvarozek J, et al. Data flow metrics in program comprehension tasks. ACM; 2018. pp. 1–6.
[69]
Krejtz K, Duchowski A, Wisiecka K, et al. Entropy of eye movements while reading code or text. ACM; 2022. pp. 8–14;.
[70]
Kuang P, Söderberg E, Niehorster D, et al. Toward gaze-assisted developer tools. IEEE; 2023. pp. 49–54.
[71]
Li X, Liu W, Wang W, et al. Assessing students’ behavior in error finding programming tests: an eye-tracking based approach. IEEE; 2019. pp. 1–6.
[72]
Li X, Liu W, Liu H, et al. Task-oriented analysis on debugging process based on eye movements and IDE interactions. IEEE; 2021. pp. 379–384.
[73]
Lin YT, Wu CC, Hou TY, et al. Tracking students’ cognitive processes during program debugging—an eye-movement approach IEEE Trans Educ 2016 59 175-186
[74]
Lin YT, Liao YZ, Hu X, et al. EEG activities during program comprehension: an exploration of cognition IEEE Access 2021 9 120407-120421
[75]
Liu L, Liu W, Li X, et al. Eye-tracking based performance analysis in error finding programming test. IEEE; 2020. pp. 477–482.
[76]
Liu L, Liu W, Li X, et al. An analysis scheme to interpret students’ cognitive process in error finding test. ACM; 2020. pp. 220–225.
[77]
Luo F, Liu R, Awoyemi ID, et al. Novel insights into elementary girls’ experiences in physiological computing. 2024. pp. 764–770.
[78]
Al Madi N. How readable is model-generated code? Examining readability and visual inspection of GitHub copilot. ACM; 2022. pp. 1–5.
[79]
Al Madi N, Peterson C, Sharif B, et al. From novice to expert: analysis of token level effects in a longitudinal eye tracking study, vol. 2021-May. IEEE; 2021. pp. 172–183.
[80]
Mansoor N, Peterson C, Dodd M, et al. Assessing the effect of programming language and task type on eye movements of computer science students ACM Trans Comput Educ 2023
[81]
McChesney I, Bond R. Gaze behaviour in computer programmers with dyslexia: considerations regarding code style, layout and crowding. ACM; 2018. pp. 1–5.
[82]
McChesney I, Bond R. Observations on the linear order of program code reading patterns in programmers with dyslexia. ACM; 2020. pp. 81–89.
[83]
McChesney I, Bond R. The effect of crowding on the reading of program code for programmers with dyslexia, vol. 2021-May. IEEE; 2021. pp. 300–310.
[84]
Melo J, Narcizo FB, Hansen DW, et al. Variability through the eyes of the programmer. IEEE; 2017. pp. 34–44;.
[85]
Molina A, Paredes M, Redondo M, et al. Assessing representation techniques of programs supported by GreedEx. IEEE; 2014. pp. 53–58.
[86]
Molina-Diaz A, Paredes-Velasco M, Redondo-Duque M, et al. Evaluation experiences of the representation techniques of greedy programs: application to the GreedEx tool, vol. 11. In: IEEE-RITA. 2016. pp. 179–86.
[87]
Nakayama M, Harada H. Eye movement features in response to comprehension performance during the reading of programs. ACM; 2020. pp. 1–5.
[88]
Nevalainen S, Sajaniemi J. An experiment on short-term effects of animated versus static visualization of operations on program perception. ACM; 2006. pp. 7–16.
[89]
Nivala M, Hauser F, Mottok J, et al. Developing visual expertise in software engineering: an eye tracking study, vol. 10–13-April-2016. IEEE; 2016. pp. 613–620.
[90]
Obaidellah U, Haek MA. Evaluating gender difference on algorithmic problems using eye-tracker. ACM; 2018. pp. 1–8.
[91]
Obaidellah U, Haek MA, and Cheng P A survey on the usage of eye-tracking in computer programming ACM Comput Surv 2018 51 1-58
[92]
Obaidellah U, Raschke M, Blascheck T. Classification of strategies for solving programming problems using AOI sequence analysis. ACM; 2019. pp. 1–9.
[93]
Obaidellah U, Blascheck T, Guarnera D, et al. A fine-grained assessment on novice programmers’ gaze patterns on pseudocode problems. ACM; 2020. pp. 1–5.
[94]
Oliveira BD, Ribeiro M, Costa JASD, et al. Atoms of confusion: the eyes do not lie. ACM; 2020. pp. 243–252.
[95]
il Park K, Sharif B. Assessing perceived sentiment in pull requests with emoji: evidence from tools and developer eye movements. IEEE; 2021. pp. 1–6.
[96]
il Park K, Weill-Tessier P, Brown N, et al. An eye tracking study assessing the impact of background styling in code editors on novice programmers’ code understanding. ACM; 2023. pp. 444–463.
[97]
Peitek N, Siegmund J, Apel S. What drives the reading order of programmers? An eye tracking study. ACM; 2020. pp. 342–353.
[98]
Peitek N, Bergum A, Rekrut M, et al. Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking study. ACM; 2022. pp. 120–131.
[99]
Peng F, Li C, Song X, et al. An eye tracking research on debugging strategies towards different types of bugs, vol. 2. IEEE; 2016. pp. 130–134.
[100]
Peterson C. Investigating the effect of polyglot programming on developers, vol. 2010-October. IEEE; 2021. pp. 1–2.
[101]
Peterson C, Abid N, Bryant C, et al. Factors influencing dwell time during source code reading—a large-scale replication experiment. ACM; 2019. pp. 1–4.
[102]
Peterson C, Halavick N, Saddler J, et al. A gaze-based exploratory study on the information seeking behavior of developers on stack overflow. ACM; 2019. pp. 1–6.
[103]
Peterson C, Saddler J, Blascheck T, et al. Visually analyzing students’ gaze on C++ code snippets. IEEE; 2019. pp. 18–25.
[104]
Peterson C, il Park K, Baysinger I, et al. An eye tracking perspective on how developers rate source code readability rules. IEEE; 2021. pp. 138–139.
[105]
Rodeghero P, McMillan C. An empirical study on the patterns of eye movement during summarization tasks, vol. 2015-November. IEEE; 2015. pp. 11–20.
[106]
Rodeghero P, McMillan C, McBurney PW, et al. Improving automated source code summarization via an eye-tracking study of programmers. ACM; 2014. pp. 390–401.
[107]
Rodeghero P, Liu C, McBurney P, et al. An eye-tracking study of java programmers and application to source code summarization IEEE Trans Softw Eng 2015 41 1038-1054
[108]
Saddler J. Looks can mean achieving: understanding eye gaze patterns of proficiency in code comprehension. ACM; 2019. pp. 1–3.
[109]
Saddler J, Peterson C, Sama S, et al. Studying developer reading behavior on stack overflow during API summarization tasks. IEEE; 2020. pp. 195–205.
[110]
Santos D, Sant’Anna C. How does feature dependency affect configurable system comprehensibility?, vol. 2019-May. IEEE; 2019. pp. 19–29.
[111]
Santos D, Sant’Anna C, Ribeiro M. An experiment on how feature dependent variables affect configurable system comprehensibility. ACM; 2023. pp. 61–70.
[112]
Santos M, Gralha C, Goulão M, et al. What is the impact of bad layout in the understandability of social goal models? IEEE; 2016. pp. 206–215.
[113]
Santos M, Gralha C, Goulão M, et al. On the impact of semantic transparency on understanding and reviewing social goal models. IEEE; 2018. pp. 228–239.
[114]
Schmedes M, Ahrens M, Nagel L, et al. Enriching vision videos with text: an eye tracking study, vol. 2022-August. IEEE; 2022. pp. 77–87.
[115]
Sepasi E, Balouchi K, Mercier J, et al. Towards a cognitive model of feature model comprehension: an exploratory study using eye-tracking, vol. A. ACM; 2022. pp. 21–31.
[116]
Sharafi Z, Soh Z, Guéhéneuc YG, et al. Women and men—different but equal: on the impact of identifier style on source code reading. IEEE; 2012. pp. 27–36.
[117]
Sharafi Z, Marchetto A, Susi A, et al. An empirical study on the efficiency of graphical vs. textual representations in requirements comprehension. IEEE; 2013. pp. 33–42.
[118]
Sharafi Z, Shaffer T, Sharif B, et al. Eye-tracking metrics in software engineering. IEEE; 2015. pp. 96–103.
[119]
Sharafi Z, Soh Z, and Guéhéneuc YG A systematic literature review on the usage of eye-tracking in software engineering Inf Softw 2015 67 79-107
[120]
Sharafi Z, Sharif B, Guéhéneuc YG, et al. A practical guide on conducting eye tracking studies in software engineering Empir Softw Eng 2020 25 3128-3174
[121]
Sharafi Z, Bertram I, Flanagan M, et al. Eyes on code: a study on developers’ code navigation strategies IEEE Trans Softw Eng 2022 48 1692-1704
[122]
Sharif B, Maletic JI. An eye tracking study on camelcase and under_score identifier styles. IEEE; 2010. pp. 196–205.
[123]
Sharif B, Maletic JI. An eye tracking study on the effects of layout in understanding the role of design patterns. IEEE; 2010. pp. 1–10.
[124]
Sharif B, Falcone M, Maletic JI. An eye-tracking study on the role of scan time in finding source code defects. ACM; 2012. pp. 381–384.
[125]
Sorg T, Abbad-Andaloussi A, Weber B. Towards a fine-grained analysis of cognitive load during program comprehension. IEEE; 2022. pp. 748–752.
[126]
Sparman S, Schulte C. Analysing the API learning process through the use of eye tracking. 2023. pp. 1–6.
[127]
Spinelli L, Pandey M, Oney S. Attention patterns for code animations: using eye trackers to evaluate dynamic code presentation techniques, vol. Part F137691. ACM; 2018. pp. 99–104.
[128]
Talsma R, Barendsen E, Smetsers S. Analyzing the influence of block highlighting on beginning programmers’ reading behavior using eye tracking. ACM; 2020. pp. 1–10.
[129]
Tobii. Pro lab user manual v 1.194, 2022.
[130]
Turner R, Falcone M, Sharif B, et al. An eye-tracking study assessing the comprehension of C++ and python source code. ACM; 2014. pp. 231–234.
[131]
Uddin MS, Gaur V, Gutwin C, et al. On the comprehension of code clone visualizations: a controlled study using eye tracking. IEEE; 2015. pp. 161–170.
[132]
Ujbanyi T, Katona J, Sziladi G, et al. Eye-tracking analysis of computer networks exam question besides different skilled groups. IEEE; 2016. pp. 277–281.
[133]
Uwano H, Nakamura M, Monden A, et al. Analyzing individual performance of source code review using reviewers’ eye movement. ACM; 2006. pp. 133–140.
[134]
Villamor M, Rodrigo M. Predicting successful collaboration in a pair programming eye tracking experiment. ACM; 2018. pp. 263–268.
[135]
Šaloun P, Malčik M, Andrešič D, et al. Using eyetracking to analyse how flowcharts are understood. IEEE; 2017. pp. 394–399.
[136]
Weber T, Winiker C, Hussmann H. A closer look at machine learning code. ACM; 2021. pp. 1–6.
[137]
Yusuf S, Kagdi H, Maletic JI. Assessing the comprehension of UML class diagrams via eye tracking. IEEE; 2007. pp. 1–10.
Index Terms
- On Eye Tracking in Software Engineering
Index terms have been assigned to the content through auto-classification.
Recommendations
A practical guide on conducting eye tracking studies in software engineering
AbstractFor several years, the software engineering research community used eye trackers to study program comprehension, bug localization, pair programming, and other software engineering tasks. Eye trackers provide researchers with insights on software ...
Eye Tracking Metrics in Software Engineering
ECSEE '18: Proceedings of the 3rd European Conference of Software Engineering EducationThis paper aims to provide an overview of the interdisciplinary combination of educational science, psychology, software engineering and the eye tracking methodology. The domain of software engineering is offering great potential for applied eye ...
Data Analysis Tools Affect Outcomes of Eye-Tracking Studies
ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementBackground: Eye-tracking studies in software engineering offer insights into the thought processes of developers and their interaction with visual information. However, the analysis of eye-tracking data lacks established standards. Particularly ...
Comments
Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 26 July 2024
Accepted: 08 June 2024
Received: 08 December 2023
Author Tags
Qualifiers
- Review-article
Funding Sources
- Bundesministerium für Bildung und Forschung
- Ostbayerische Technische Hochschule Regensburg (3398)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 01 Jan 2025