Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3387904.3389279acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

What Drives the Reading Order of Programmers?: An Eye Tracking Study

Published: 12 September 2020 Publication History

Abstract

Background: The way how programmers comprehend source code depends on several factors, including the source code itself and the programmer. Recent studies showed that novice programmers tend to read source code more like natural language text, whereas experts tend to follow the program execution flow. But, it is unknown how the linearity of source code and the comprehension strategy influence programmers' linearity of reading order.
Objective: We replicate two previous studies with the aim of additionally providing empirical evidence on the influencing effects of linearity of source code and programmers' comprehension strategy on linearity of reading order.
Methods: To understand the effects of linearity of source code on reading order, we conducted a non-exact replication of studies by Busjahn et al. and Peachock et al., which compared the reading order of novice and expert programmers. Like the original studies, we used an eye-tracker to record the eye movements of participants (12 novice and 19 intermediate programmers).
Results: In line with Busjahn et al. (but different from Peachock et al.), we found that experience modulates the reading behavior of participants. However, the linearity of source code has an even stronger effect on reading order than experience, whereas the comprehension strategy has a minor effect.
Implications: Our results demonstrate that studies on the reading behavior of programmers must carefully select source code snippets to control the influence of confounding factors. Furthermore, we identify a need for further studies on how programmers should structure source code to align it with their natural reading behavior to ease program comprehension.

References

[1]
Nahla Abid, Bonita Sharif, Natalia Dragan, Hend Alrasheed, and Jonathan Maletic. 2019. Developer Reading Behavior While Summarizing Java Methods: Size and Context Matters. In Proc. Int. Conf. Software Engineering (ICSE). IEEE, 384--395.
[2]
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1--48.
[3]
Jennifer Bauer, Janet Siegmund, Norman Peitek, Johannes Hofmeister, and Sven Apel. 2019. Indentation: Simply a Matter of Style or Support for Program Comprehension?. In Proc. Int. Conf. Program Comprehension (ICPC). IEEE, 154--164.
[4]
Roman Bednarik. 2012. Expertise-Dependent Visual Attention Strategies Develop over Time during Debugging with Multiple Code Representations. Int'l Journal of Human-Computer Studies 70, 2 (2012), 143--155.
[5]
Tanya Beelders and Jean-Pierre du Plessis. 2016. The Influence of Syntax Highlighting on Scanning and Reading Behaviour for Source Code. In Proc. Annual Conf. of the South African Institute of Computer Scientists and Information Technologists (SAICSIT). ACM, 1--10.
[6]
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57, 1 (1995), 289--300.
[7]
Dave Binkley, Marcia Davis, Dawn Lawrie, Jonathan Maletic, Christopher Morrell, and Bonita Sharif. 2013. The Impact of Identifier Style on Effort and Comprehension. Empirical Software Engineering 18, 2 (2013), 219--276.
[8]
David Binkley, Marcia Davis, Dawn Lawrie, and Christopher Morrell. 2009. To CamelCase or Under_score. In Proc. Int. Conf. Program Comprehension (ICPC). IEEE, 158--167.
[9]
Tanja Blascheck and Bonita Sharif. 2019. Visually Analyzing Eye Movements on Natural Language Texts and Source Code Snippets. In Proc. Symposium on Eye Tracking Research & Applications (ETRA). ACM, 14:1-14:9.
[10]
Ruven Brooks. 1983. Towards a Theory of the Comprehension of Computer Programs. Int. Journal of Man-Machine Studies 18, 6 (1983), 543--554.
[11]
Raymond Buse and Westley Weimer. 2008. A Metric for Software Readability. In Proc. Int'l Symposium on Software Testing and Analysis. ACM, 121--130.
[12]
Raymond Buse and Westley Weimer. 2010. Learning a Metric for Code Readability. IEEE Transactions on Software Engineering 36, 4 (2010), 546--558.
[13]
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. In Proc. Int. Conf. Program Comprehension (ICPC). IEEE, 255--265.
[14]
Teresa Busjahn, Carsten Schulte, Bonita Sharif, Andrew Begel, Michael Hansen, Roman Bednarik, Paul Orlov, Petri Ihantola, Galina Shchekotova, and Maria Antropova. 2014. Eye Tracking in Computing Education. In Proc. Conf. on International Computing Education Research. ACM, 3--10.
[15]
Don Coleman, Dan Ash, Bruce Lowther, and Paul Oman. 1994. Using Metrics to Evaluate Software System Maintainability. Computer 27, 8 (1994), 44--49.
[16]
Filipe Cristino, Sebastiaan Mathôt, Jan Theeuwes, and Iain Gilchrist. 2010. Scan-Match: A Novel Method for Comparing Fixation Sequences. Behavior Research Methods 42, 3 (2010), 692--700.
[17]
Martha Crosby and Jan Stelovsky. 1990. How Do We Read Algorithms? A Case Study. Computer 23, 1 (1990), 25--35.
[18]
Hubert Dreyfus and Stuart Dreyfus. 1986. Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. The Free Press (1986).
[19]
John Fox and Sanford Weisberg. 2019. An R Companion to Applied Regression (third ed.). Sage, Thousand Oaks CA.
[20]
Yorai Geffen and Shahar Maoz. 2016. On Method Ordering. In Proc. Int. Conf. Program Comprehension (ICPC). IEEE, 1--10.
[21]
David Gilmore and Thomas Green. 1988. Programming Plans and Programming Expertise. The Quarterly Journal of Experimental Psychology 40, 3 (1988), 423--442.
[22]
Kenneth Holmqvist, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost Van de Weijer. 2011. Eye Tracking: A Comprehensive Guide to Methods and Measures. OUP Oxford.
[23]
Yoshiharu Ikutani, Takatomi Kubo, Satoshi Nishida, Hideaki Hata, Kenichi Matsumoto, Kazushi Ikeda, and Shinji Nishimoto. 2020. Expert Programmers have Fine-Tuned Cortical Representations of Source Code. bioRxiv 2020.01.28.923953 (2020).
[24]
Yoshiharu Ikutani and Hidetake Uwano. 2014. Brain Activity Measurement during Program Comprehension with NIRS. In Proc. Int. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD). IEEE, 1--6.
[25]
Ahmad Jbara and Dror Feitelson. 2017. How Programmers Read Regular Code: A Controlled Experiment Using Eye Tracking. Empirical Software Engineering 22, 3 (2017), 1440--1477.
[26]
Thomas LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining Mental Models: A Study of Developer Work Habits. In Proc. Int. Conf. Software Engineering (ICSE). ACM, 492--501.
[27]
Seolhwa Lee, Danial Hooshyar, Hyesung Ji, Kichun Nam, and Heuiseok Lim. 2018. Mining Biometric Data to Predict Programmer Expertise and Task Difficulty. Cluster Computing 21, 1 (2018), 1097--1107.
[28]
SeolHwa Lee, Andrew Matteson, Danial Hooshyar, SongHyun Kim, JaeBum Jung, GiChun Nam, and HeuiSeok Lim. 2016. Comparing Programming Language Comprehension between Novice and Expert Programmers Using EEG Analysis. In Proc. Int. Conf. Bioinformatics and Bioengineering (BIBE). IEEE, 350--355.
[29]
Stefan Lessmann, Bart Baesens, Christophe Mues, and Swantje Pietsch. 2008. Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings. IEEE Transactions on Software Engineering 34, 4 (2008), 485--496.
[30]
Mary Lindstrom and Douglas Bates. 1988. Newton-Raphson and EM Algorithms for Linear Mixed-Effects Models for Repeated-Measures Data. Journal of the American Statistical Association 83, 404 (1988), 1014--1022.
[31]
Steve McConnell. 2011. What Does 10x Mean? Measuring Variations in Programmer Productivity. Making Software, O'Reilly 16 (2011).
[32]
Jerry Mead, Simon Gray, John Hamer, Richard James, Juha Sorva, Caroline St Clair, and Lynda Thomas. 2006. A Cognitive Approach to Identifying Measurable Milestones for Programming Skill Acquisition. ACM SIGCSE Bulletin 38, 4 (2006), 182--194.
[33]
Takao Nakagawa, Yasutaka Kamei, Hidetake Uwano, Akito Monden, Kenichi Matsumoto, and Daniel M. German. 2014. Quantifying Programmers' Mental Workload During Program Comprehension Based on Cerebral Blood Flow Measurement: A Controlled Experiment. In Proc. Int. Conf. Software Engineering (ICSE). ACM, 448--451.
[34]
Saul Needleman and Christian Wunsch. 1970. A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins. Journal of Molecular Biology 48, 3 (1970), 443--453.
[35]
Marcus Nyström and Kenneth Holmqvist. 2010. An Adaptive Algorithm for Fixation, Saccade, and Glissade Detection in Eyetracking Data. Behavior Research Methods 42, 1 (2010), 188--204.
[36]
Pavel Orlov. 2017. Ambient and Focal Attention During Source-Code Comprehension. In Proc. Int'l Workshop on Eye Movements in Programming (EMIP). 12--13.
[37]
Christopher Palmer and Bonita Sharif. 2016. Towards Automating Fixation Correction for Source Code. In Proc. Symposium on Eye Tracking Research & Applications (ETRA). ACM, 65--68.
[38]
Patrick Peachock, Nicholas Iovino, and Bonita Sharif. 2017. Investigating Eye Movements in Natural Language and C++ Source Code - A Replication Experiment. In Augmented Cognition. Neurocognition and Machine Learning. Springer International Publishing, 206--218.
[39]
Nancy Pennington. 1987. Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs. Cognitive Psychology 19, 3 (1987), 295--341.
[40]
Cole Peterson, Jonathan Saddler, Tanja Blascheck, and Bonita Sharif. 2019. Visually Analyzing Students' Gaze on C++ Code Snippets. In Proc. Int'l Workshop on Eye Movements in Programming (EMIP). IEEE, 18--25.
[41]
Paige Rodeghero and Collin McMillan. 2015. An Empirical Study on the Patterns of Eye Movement During Summarization Tasks. In Proc. Int. Symp. Empirical Software Engineering and Measurement (ESEM). IEEE, 1--10.
[42]
Samuel Shapiro and Martin Wilk. 1965. An Analysis of Variance Test for Normality (Complete Samples). Biometrika 52, 3/4 (1965), 591--611.
[43]
Bonita Sharif, Michael Falcone, and Jonathan Maletic. 2012. An Eye-Tracking Study on the Role of Scan Time in Finding Source Code Defects. In Proc. Symposium on Eye Tracking Research and Applications (ETRA). ACM, 381--384.
[44]
Bonita Sharif and Johnathon Maletic. 2010. An Eye Tracking Study on camel Case and under_score Identifier Styles. In Proc. Int. Conf. Program Comprehension (ICPC). IEEE, 196--205.
[45]
Ben Shneiderman. 1977. Measuring Computer Program Quality and Comprehension. Int. J. Man-Machine Studies 9, 4 (1977), 465--478.
[46]
Ben Shneiderman and Richard Mayer. 1979. Syntactic/Semantic Interactions in Programmer Behavior: A Model and Experimental Results. Int. J. Parallel Programming 8, 3 (1979), 219--238.
[47]
Janet Siegmund. 2016. Program Comprehension: Past, Present, and Future. In Proc. Int. Conf. Software Analysis, Evolution, and Reengineering (SANER). IEEE, 13--20.
[48]
Janet Siegmund, Christian Kästner, Sven Apel, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2014. Understanding Understanding Source Code with Functional Magnetic Resonance Imaging. In Proc. Int. Conf. Software Engineering (ICSE). ACM, 378--389.
[49]
Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2014. Measuring and Modeling Programming Experience. Empirical Software Engineering 19, 5 (2014), 1299--1334.
[50]
Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister, Christian Kästner, Andrew Begel, Anja Bethmann, and André Brechmann. 2017. Measuring Neural Efficiency of Program Comprehension. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 140--150.
[51]
Elliot Soloway and Kate Ehrlich. 1984. Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering 10, 5 (1984), 595--609.
[52]
Rebecca Tiarks. 2011. What Programmers Really Do: An Observational Study. Softwaretechnik-Trends 31, 2 (2011), 36--37.
[53]
Rachel Turner, Michael Falcone, Bonita Sharif, and Alina Lazar. 2014. An Eye-Tracking Study Assessing the Comprehension of C + + and Python Source Code. In Proc. Symposium on Eye Tracking Research and Applications (ETRA). ACM, 231--234.
[54]
Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2006. Analyzing Individual Performance of Source Code Review Using Reviewers' Eye Movement. In Proc. Symposium on Eye Tracking Research & Applications (ETRA). ACM, 133--140.
[55]
Alberto Salvador Núñez Varela, Hector G. Pérez-González, Francisco E. Martínez-Perez, and Carlos Soubervielle-Montalvo. 2017. Source Code Metrics: A Systematic Mapping Study. Journal of Systems and Software 128 (2017), 164--197.

Cited By

View all
  • (2024)An Investigation of How Software Developers Read Machine Learning CodeProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686678(165-176)Online publication date: 24-Oct-2024
  • (2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
  • (2024)Attention Dynamics in Programming: Eye Gaze Patterns of High- vs. Low-Ability Novice CodersProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3655095(1-6)Online publication date: 4-Jun-2024
  • Show More Cited By

Index Terms

  1. What Drives the Reading Order of Programmers?: An Eye Tracking Study

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICPC '20: Proceedings of the 28th International Conference on Program Comprehension
      July 2020
      481 pages
      ISBN:9781450379588
      DOI:10.1145/3387904
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 12 September 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. eye tracking
      2. linearity of reading order
      3. program comprehension
      4. programmer experience
      5. source code linearity

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      Conference

      ICPC '20
      Sponsor:

      Upcoming Conference

      ICSE 2025

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)95
      • Downloads (Last 6 weeks)16
      Reflects downloads up to 09 Nov 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)An Investigation of How Software Developers Read Machine Learning CodeProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686678(165-176)Online publication date: 24-Oct-2024
      • (2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
      • (2024)Attention Dynamics in Programming: Eye Gaze Patterns of High- vs. Low-Ability Novice CodersProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3655095(1-6)Online publication date: 4-Jun-2024
      • (2024)Exploring the Impact of Source Code Linearity on the Programmers' Comprehension of API Code ExamplesProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644395(236-240)Online publication date: 15-Apr-2024
      • (2024)EyeTrans: Merging Human and Machine Attention for Neural Code SummarizationProceedings of the ACM on Software Engineering10.1145/36437321:FSE(115-136)Online publication date: 12-Jul-2024
      • (2024)Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent VariablesProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648213(42-47)Online publication date: 16-Apr-2024
      • (2024)Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted ProgrammingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641936(1-16)Online publication date: 11-May-2024
      • (2024)A Study on the Pythonic Functional Constructs' UnderstandabilityProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639211(1-13)Online publication date: 20-May-2024
      • (2024)Towards Generating Maintainable and Comprehensible API Code Examples2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00090(830-834)Online publication date: 12-Mar-2024
      • (2024)On current limitations of online eye-tracking to study the visual processing of source codeInformation and Software Technology10.1016/j.infsof.2024.107502174:COnline publication date: 1-Oct-2024
      • Show More Cited By

      View Options

      Get Access

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media