research-article

A replication study on code comprehension and expertise using lightweight biometric sensors

Authors:

Daniela Girardi,

Nicole Novielli,

Luigi Quaranta,

Filippo LanubileAuthors Info & Claims

ICPC '19: Proceedings of the 27th International Conference on Program Comprehension

Pages 311 - 322

https://doi.org/10.1109/ICPC.2019.00050

Published: 25 May 2019 Publication History

Abstract

Code comprehension has been recently investigated from physiological and cognitive perspectives using medical imaging devices. Floyd et al. (i.e., the original study) used fMRI to classify the type of comprehension tasks performed by developers and relate their results to their expertise. We replicate the original study using lightweight biometrics sensors. Our study participants---28 undergrads in computer science---performed comprehension tasks on source code and natural language prose. We developed machine learning models to automatically identify what kind of tasks developers are working on leveraging their brain-, heart-, and skin-related signals. The best improvement over the original study performance is achieved using solely the heart signal obtained through a single device (BAC 87% vs. 79.1%). Differently from the original study, we did not observe a correlation between the participants' expertise and the classifier performance (τ = 0.16, p = 0.31). Our findings show that lightweight biometric sensors can be used to accurately recognize comprehension tasks opening interesting scenarios for research and practice.

References

[1]

M. Ciolkowski, O. Laitenberger, and S. Biffl, "Software Reviews: The State of the Practice," IEEE software, no. 6, pp. 46--51, 2003.

Digital Library

[2]

F. Lanubile, T. Mallardo, F. Calefato, C. Denger, and M. Ciolkowski, "Assessing the Impact of Active Guidance for Defect Detection: A Replicated Experiment," in null. IEEE, 2004, pp. 269--279.

Digital Library

[3]

D. Rombach, M. Ciolkowski, R. Jeffery, O. Laitenberger, F. McGarry, and F. Shull, "Impact of Research on Practice in the Field of Inspections, Reviews and Walkthroughs: Learning from Successful Industrial Uses," ACM SIGSOFT Software Engineering Notes, vol. 33, no. 6, pp. 26--35, 2008.

Digital Library

[4]

K. Nishizono, S. Morisakl, R. Vivanco, and K. Matsumoto, "Source Code Comprehension Strategies and Metrics to Predict Comprehension Effort in Software Maintenance and Evolution Tasks-An Empirical Study with Industry Practitioners," in Software Maintenance (ICSM), 2011 27th IEEE International Conference on. IEEE, 2011, pp. 473--481.

Digital Library

[5]

A. Armaly, P. Rodeghero, and C. McMillan, "AudioHighlight: Code Skimming for Blind Programmers," in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2018, pp. 206--216.

[6]

J. Siegmund, C. Kästner, S. Apel, C. Parnin, A. Bethmann, T. Leich, G. Saake, and A. Brechmann, "Understanding Understanding Source Code with Functional Magnetic Resonance Imaging," in Proceedings of the 36th International Conference on Software Engineering. ACM, 2014, pp. 378--389.

Digital Library

[7]

B. Floyd, T. Santander, and W. Weimer, "Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise," in Proceedings of the 39th International Conference on Software Engineering. IEEE Press, 2017, pp. 175--186.

Digital Library

[8]

F. Ebert, F. Castor, N. Novielli, and A. Serebrenik, "Confusion in Code Reviews: Reasons, Impacts, and Coping Strategies," in Proceedings of 26th International Conference on Software Analysis, Evolution and Reengineering SANER 2019. IEEE Press, 2019, pp. 49--60.

[9]

M. Züger, C. Corley, A. N. Meyer, B. Li, T. Fritz, D. Shepherd, V. Augustine, P. Francis, N. Kraft, and W. Snipes, "Reducing Interruptions at Work: A Large-scale Field Study of FlowLight," in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2017, pp. 61--72.

Digital Library

[10]

T. Fritz, A. Begel, S. C. Müller, S. Yigit-Elliott, and M. Züger, "Using Psycho-physiological Measures to Assess Task Difficulty in Software Development," in Proceedings of the 36th International Conference on Software Engineering. ACM, 2014, pp. 402--413.

Digital Library

[11]

S. Fakhoury, Y. Ma, V. Arnaoudova, and O. Adesope, "The Effect of Poor Source Code Lexicon and Readability on Developers' Cognitive Load," in Proc. Int'l Conf. Program Comprehension (ICPC), 2018.

Digital Library

[12]

D. W. Rowe, J. Sibert, and D. Irwin, "Heart Rate Variability: Indicator of User State as an Aid to Human-computer Interaction," in Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 1998, pp. 480--487.

Digital Library

[13]

M. Gjoreski, M. Luštrek, and V. Pejović, "My Watch Says I'm Busy: Inferring Cognitive Load with Low-Cost Wearables," in Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. ACM, 2018, pp. 1234--1240.

Digital Library

[14]

J. C. Carver, "Towards Reporting Guidelines for Experimental Replications: A Proposal," in 1st International Workshop on Replication in Empirical Software Engineering Research, 2010.

[15]

M. T. Baldassarre, J. Carver, O. Dieste, and N. Juristo, "Replication Types: Towards a Shared Taxonomy," in Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. ACM, 2014, p. 18.

Digital Library

[16]

F. J. Shull, J. C. Carver, S. Vegas, and N. Juristo, "The Role of Replications in Empirical Software Engineering," Empirical software engineering, vol. 13, no. 2, pp. 211--218, 2008.

Digital Library

[17]

N. Peitek, J. Siegmund, S. Apel, C. Kästner, C. Parnin, A. Bethmann, T. Leich, G. Saake, and A. Brechmann, "A Look into Programmers' Heads," IEEE Transactions on Software Engineering, 2018.

[18]

S. Radevski, H. Hata, and K. Matsumoto, "Real-time Monitoring of Neural State in Assessing and Improving Software Developers' Productivity," in Proceedings of the Eighth International Workshop on Cooperative and Human Aspects of Software Engineering. IEEE Press, 2015, pp. 93--96.

Digital Library

[19]

S. Müller and T. Fritz, "Stuck and Frustrated or in Flow and Happy: Sensing Developers' Emotions and Progress," in Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on, vol. 1. IEEE, 2015, pp. 688--699.

Digital Library

[20]

C. Parnin, "Subvocalization-Toward Hearing the Inner Thoughts of Developers," in Program Comprehension (ICPC), 2011 IEEE 19th International Conference on. IEEE, 2011, pp. 197--200.

Digital Library

[21]

S. Müller and T. Fritz, "Using (bio) Metrics to Predict Code Quality Online," in Proceedings of the 38th International Conference on Software Engineering. ACM, 2016, pp. 452--463.

Digital Library

[22]

Y. Ikutani and H. Uwano, "Brain Activity Measurement During Program Comprehension with NIRS," in Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 2014 15th IEEE/ACIS International Conference on. IEEE, 2014, pp. 1--6.

[23]

N. Peitek, J. Siegmund, C. Parnin, S. Apel, J. Hofmeister, and A. Brechmann, "Simultaneous Measurement of Program Comprehension with fMRI and Eye Tracking: A Case Study," in Proc. Intl Symposium Empirical Software Engineering and Measurement (ESEM). ACM, 2018.

Digital Library

[24]

J. J. Braithwaite, D. G. Watson, R. Jones, and M. Rowe, "A Guide for Analysing Electrodermal Activity (EDA) & Skin Conductance Responses (SCRs) for Psychological Experiments," Psychophysiology, vol. 49, no. 1, pp. 1017--1034, 2013.

[25]

F. Canento, A. Fred, H. Silva, H. Gamboa, and A. Loureno, "Multimodal biosignal sensor data handling for emotion recognition," in SENSORS, 2011 IEEE, 2011, pp. 647--650.

[26]

A. Greco, G. Valenza, A. Lanata, E. P. Scilingo, and L. Citi, "cvxEDA: A Convex Optimization Approach to Electrodermal Activity Processing," IEEE Transactions on Biomedical Engineering, vol. 63, no. 4, pp. 797--804, 2016.

[27]

H. Trevor, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, 2009.

[28]

S. Koelstra, C. Mühl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Yiannis) Patras, "Deap: A database for emotion analysis using physiological signals," IEEE Transactions on Affective Computing, vol. 3, pp. 18--31, 12 2011.

Digital Library

[29]

C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, "Automated parameter optimization of classification techniques for defect prediction models," in 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 2016, pp. 321--332.

Digital Library

[30]

C. Tantithamthavorn, S. McIntosh, "The impact of automated parameter optimization on defect prediction models," IEEE Transactions on Software Engineering, 2018.

[31]

J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, "Algorithms for Hyper-parameter Optimization," in Advances in neural information processing systems, 2011, pp. 2546--2554.

Digital Library

[32]

M. Kuhn, "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, Articles, vol. 28, no. 5, pp. 1--26, 2008.

[33]

C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, "An Empirical Comparison of Model Validation Techniques for Defect Prediction Models," IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 1--18, 2017.

Digital Library

[34]

F. Sebastiani, "Machine learning in automated text categorization," ACM Comput. Surv., vol. 34, no. 1, pp. 1--47, Mar. 2002.

Digital Library

[35]

J. Sillito, G. C. Murphy, and K. De Volder, "Questions Programmers ask During Software Evolution Tasks," in Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering. ACM, 2006, pp. 23--34.

Digital Library

[36]

J. Hatcher, M. J. Snowling, and Y. M. Griffiths, "Cognitive Assessment of Dyslexic Students in Higher Education," British journal of educational psychology, vol. 72, no. 1, pp. 119--133, 2002.

[37]

E. Macaro, "Strategies for Language Learning and For Language Use: Revising the Theoretical Framework," The Modern Language Journal, vol. 90, no. 3, pp. 320--337, 2006.

[38]

W. A. Grove, T. Wasserman, and A. Grodner, "Choosing a Proxy for Academic Aptitude," The Journal of Economic Education, vol. 37, no. 2, pp. 131--147, 2006.

[39]

G. E. Noether, "Why Kendall Tau?" Teaching Statistics, vol. 3, no. 2, pp. 41--43, 1981.

[40]

W. S. Humphrey, "Personal Software Process (PSP)," Encyclopedia of Software Engineering, 2002.

[41]

X. Yang, R. G. Kula, N. Yoshida, and H. Iida, "Mining the Modern Code Review Repositories: A Dataset of People, Process and Product," in Proceedings of the 13th International Conference on Mining Software Repositories. ACM, 2016, pp. 460--463.

Digital Library

[42]

K. Kevic, B. Walters, T. Shaffer, B. Sharif, D. C. Shepherd, and T. Fritz, "Eye Gaze and Interaction Contexts for Change Tasks-Observations and Potential," Journal of Systems and Software, vol. 128, pp. 252--266, 2017.

Digital Library

[43]

B. Sharif, T. Shaffer, J. Wise, and J. I. Maletic, "Tracking Developers' Eyes in the IDE," IEEE Software, vol. 33, no. 3, pp. 105--108, 2016.

Digital Library

[44]

M. P. Robillard, A. Marcus, C. Treude, G. Bavota, O. Chaparro, N. Ernst, M. A. Gerosa, M. Godfrey, M. Lanza, and M. Linares-Vásquez, "On-demand Developer Documentation," in Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 2017, pp. 479--483.

[45]

A. Dunsmore and M. Roper, "A Comparative Evaluation of Program Comprehension Measures," The Journal of Systems and Software, vol. 52, no. 3, pp. 121--129, 2000.

Digital Library

[46]

T. M. Shaft and I. Vessey, "The Relevance of Application Domain Knowledge: The Case of Computer Program Comprehension," Information systems research, vol. 6, no. 3, pp. 286--299, 1995.

Digital Library

[47]

J. Feigenspan, M. Schulze, M. Papendieck, C. Kästner, R. Dachselt, V. Köppen, M. Frisch, and G. Saake, "Supporting Program Comprehension in Large Preprocessor-based Software Product Lines," IET software, vol. 6, no. 6, pp. 488--501, 2012.

[48]

A. Van Deursen, "Program Comprehension Risks and Opportunities in Extreme Programming," in Reverse Engineering, 2001. Proceedings. Eighth Working Conference on. IEEE, 2001, pp. 176--185.

Digital Library

Cited By

Karas ZBansal AZhang YLi TMcMillan CHuang Y(2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664808
Wyrich MBogner JWagner S(2023)40 Years of Designing Code Comprehension Experiments: A Systematic Mapping StudyACM Computing Surveys10.1145/362652256:4(1-42)Online publication date: 9-Nov-2023
https://dl.acm.org/doi/10.1145/3626522
Feldman KKellogg MChaparro OChandra SBlincoe KTonella P(2023)On the Relationship between Code Verifiability and UnderstandabilityProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616242(211-223)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616242
Show More Cited By

Recommendations

Embodied comprehension of stories: Interactions between language regions and modality-specific neural systems

The embodied view of language processing proposes that comprehension involves multimodal simulations, a process that retrieves a comprehender's perceptual, motor, and affective knowledge through reactivation of the neural systems responsible for ...
Anti-spoofing study on palm biometric features
Highlights
- A dual-wavelength synchronous acquisition system is proposed.
- Static (palmprint, vein) and dynamic (SpO2, pulse) biometrics acquired at same time.
- Most multispectrum systems are upgradeable for liveness info with minimal ...
Abstract
Spoofing attacks severely threaten the security of biometric recognition systems. Two concerns are raised regarding palm biometrics. First, the anti-spoofing ability of the palm vein as a subcutaneous feature needs further exploration. In-depth ...
The neurobiology of agrammatic sentence comprehension: A lesion study

Broca's area has long been implicated in sentence comprehension. Damage to this region is thought to be the central source of "agrammatic comprehension" in which performance is substantially worse and near chance on sentences with noncanonical word ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '19: Proceedings of the 27th International Conference on Program Comprehension

May 2019

400 pages

General Chair:
Yann-Gaël Guéhéneuc
Concordia University, Montréal, Canada
,
Program Chairs:
Foutse Khomh
Polytechnique Montréal, Canada
,
Federica Sarro
University College London, London, United Kingdom

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 25 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 25, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
86
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Karas ZBansal AZhang YLi TMcMillan CHuang Y(2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664808
Wyrich MBogner JWagner S(2023)40 Years of Designing Code Comprehension Experiments: A Systematic Mapping StudyACM Computing Surveys10.1145/362652256:4(1-42)Online publication date: 9-Nov-2023
https://dl.acm.org/doi/10.1145/3626522
Feldman KKellogg MChaparro OChandra SBlincoe KTonella P(2023)On the Relationship between Code Verifiability and UnderstandabilityProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616242(211-223)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616242
Vieira RFarias K(2020)Usage of Psychophysiological Data as an Improvement in the Context of Software Engineering: A Systematic Mapping StudyProceedings of the XVI Brazilian Symposium on Information Systems10.1145/3411564.3411580(1-8)Online publication date: 3-Nov-2020
https://dl.acm.org/doi/10.1145/3411564.3411580
Muñoz Barón MWyrich MWagner S(2020)An Empirical Validation of Cognitive Complexity as a Measure of Source Code UnderstandabilityProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3410636(1-12)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3410636
Girardi DNovielli NFucci DLanubile FRothermel GBae D(2020)Recognizing developers' emotions while programmingProceedings of the ACM/IEEE 42nd International Conference on Software Engineering10.1145/3377811.3380374(666-677)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377811.3380374

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents