article

Confounding parameters on program comprehension: a literature survey

Authors:

Janet Siegmund,

Jana SchumannAuthors Info & Claims

Empirical Software Engineering, Volume 20, Issue 4

Pages 1159 - 1192

https://doi.org/10.1007/s10664-014-9318-8

Published: 01 August 2015 Publication History

Abstract

Program comprehension is an important human factor in software engineering. To measure and evaluate program comprehension, researchers typically conduct experiments. However, designing experiments requires considerable effort, because confounding parameters need to be controlled for. Our aim is to support researchers in identifying relevant confounding parameters and select appropriate techniques to control their influence. To this end, we conducted a literature survey of 13 journals and conferences over a time span of 10 years. As result, we created a catalog of 39 confounding parameters, including an overview of measurement and control techniques. With the catalog, we give experimenters a tool to design reliable and valid experiments.

References

[1]

Anderson M (2001) Permutation tests for univariate or multivariate analysis of variance and regression. Can J Fish Aquat Sci 58(3):626-639

[2]

Anderson T, Finn J (1996) The new statistical analysis of data. Springer, New York

[3]

Baddeley A (2001) Is working memory still working? Am Psychol 56(11):851-864

[4]

Beckwith L, Burnett M, Wiedenbeck S, Cook C, Sorte S, Hastings M (2005) Effectiveness of end-user debugging software features: are there gender issues? In: Proc. Conf. Human Factors in Computing Systems (CHI), pp. 869-878. ACM Press

[5]

Bergersen G, Gustafsson JE (2011) Programming skill, knowledge, and working memory among professional software developers from an investment theory perspective. J Individ Differ 32(4):201-209

[6]

Bettenburg N, Hassan A (2010) Studying the impact of social structures on software quality. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 124-133. IEEE CS

[7]

Biffl S, Halling M (2003) Investigating the defect detection effectiveness and cost benefit of nominal inspection teams. IEEE Trans Softw Eng 29(5):385-397

Digital Library

[8]

Binkley D, Lawrie D, Maex S, Morrell C (2008) Impact of limited memory resources. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 83-92. IEEE Computer Society

[9]

Boehm B (1981) Software engineering economics. Prentice Hall, Englewood Cliffs

[10]

Briand LC, Labiche Y, Di Penta M, Yan-Bondoc HD (2005) An experimental investigation of formality in UML-based development. IEEE Trans Softw Eng 31(10):833-849

Digital Library

[11]

Brooks R (1978) Using a behavioral theory of program comprehension in software engineering. In: Proc. Int'l Conf. Software Engineering (ICSE), pp. 196-201. IEEE CS

[12]

Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37-46

[13]

Cook T, Campbell D (1979) Quasi-experimentation: design & analysis issues for field settings. Houghton Mifflin, Boston

[14]

Corbett A, Anderson J (2001) Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In: Proc. Conf. Human Factors in Computing Systems (CHI), pp. 245-252. ACM Press

[15]

Druin A, Foss E, Hutchinson H, Golub E, Hatley L (2010) Children's roles using keyword search interfaces at home. In: Proc. Conf. Human Factors in Computing Systems (CHI), pp. 413-422. ACM Press

[16]

Dybå T, Kampenes VB, Sjøberg D (2006) A systematic review of statistical power in software engineering experiments. J Inf Softw Technol 48(8):745-755

[17]

Dzidek W, Arisholm E, Briand L (2008) A Realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34(3):407-432

Digital Library

[18]

Ellis B, Stylos J, Myers B (2007) The factory pattern in API design: a usability evaluation. In: Proc. Int'l Conf. Software Engineering (ICSE), pp. 302-312. IEEE CS

[19]

Ericsson K, Simon H (1980) Verbal reports as data. Psychol Rev 87(2):215-251

[20]

Feigenspan J (2009) Empirical comparison of FOSD approaches regarding program comprehension¿a feasibility study. Master's thesis, University of Magdeburg

[21]

Feigenspan J, Siegmund N, Fruth J (2011) On the role of program comprehension in embedded systems. In: Proc. Workshop Software Reengineering (WSR), pp. 34-35. http://wwwiti.cs.uni-magdeburg.de/iti_db/publikationen/ps/auto/FeSiFr11

[22]

Feigenspan J, Kästner C, Liebig J, Apel S, Hanenberg S (2012) Measuring programming experience. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 73-82. IEEE CS

[23]

Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32(200):675-701

[24]

Fry Z, Weimer W (2010) A human study of fault localization accuracy. In: Proc. Int'l Conf. Software Maintenance (ICSM), pp. 1-10. IEEE CS

[25]

Goldstein B (2002) Sensation and perception, fifth edn. Cengage Learning Services

[26]

Gong L, Lai J (2001) Shall we mix synthetic speech and human speech impact on users' performance, perception, and attitude. In: Proc. Conf. Human Factors in Computing Systems (CHI), pp. 158-165. ACM Press (2001)

[27]

Goodwin J (1999) Research in psychology: methods and design, 2nd edn. Wiley

[28]

Grigoreanu V, Cao J, Kulesza T, Bogart C, Rector K, Burnett M, Wiedenbeck S (2008) Can feature design reduce the gender gap in end-user software development environments? In: Proc. Symposium Visual Languages and Human-Centric Computing (VLHCC), pp. 149-156. IEEE CS

[29]

Güle¿ir G, Berg K, Bergmans L, Ak¿it M (2009) Experimental evaluation of a tool for the verification and transformation of source code in event-driven systems. Empir Softw Eng 14(6):720-777

[30]

Hu W, Lee H, Zhang Q, Liu T, Geng L, Seghier M, Shakeshaft C, Twomey T, Green D, Yang Y, Price C (2010) Developmental dyslexia in Chinese and English populations: dissociating the effect of dyslexia from language differences. Brain 133(6):1694-1706

[31]

Ishihara S (1972) Test for colour-blindness. Kanehara Shuppan Co., Tokyo

[32]

Jablonski P, Hou D (2010) Aiding software maintenance with copy-and-paste clone-awareness. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 170-179. IEEE CS

[33]

Jäger A, Süß HM, Beauducel A (1997) Berliner Intelligenzstruktur-Test. Hogrefe, Göttingen

[34]

Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to advanced empirical software engineering, pp. 201-228. Springer

[35]

Jensen E (1998) Teaching with the brain in mind. Atlantic Books, London

[36]

Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer, Boston

[37]

Kampenes V, Dybå T, Hannay J, Sjøberg D (2009) A systematic review of quasi-experiments in software engineering. Inf Softw Technol 51(1):71-82

Digital Library

[38]

Ko A, Uttl B (2003) Individual differences in program comprehension strategies in unfamiliar programming systems. In: Proc. Int'l Workshop Program Comprehension (IWPC), pp. 175-184. IEEE CS

[39]

Ko A, Myers B, Coblenz M, Aung H (2006) An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans Softw Eng 32(12):971-987

Digital Library

[40]

McConnell S (2011) What does 10× mean? Measuring variations in programmer productivity. In: Making Software, pp. 567-574. O'Reilly & Associates, Inc

[41]

McQuiggan SW, Rowe JP, Lester JC (2008) The effects of empathetic virtual characters on presence in narrative-centered learning environments. In: Proc. Conf. Human Factors in Computing Systems (CHI), pp. 1511-1520. ACM Press

[42]

Miller G (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev 63(2):81-97

[43]

Mook D (1996) Motivation: the organization of action, 2nd edn. W.W. Norton & Co., New York

[44]

Neumann Jv (1945) First draft of a report on the EDVAC

[45]

Oberauer K, Süß HM, Schulze R, Wilhelm O, Wittmann W (2000) Working memory capacity¿facets of a cognitive ability construct. Personal Individ Differ 29(6):1017-1045

[46]

Oezbek C, Prechelt L (2007) Jtourbus: simplifying program understanding by documentation that provides tours through the source code. In: Proc. Int' l Conf. Software Maintenance (ICSM), pp. 64-73. IEEE CS

[47]

Pennington N (1987) Stimulus structures and mental representations in expert comprehension of computer programs. Cogn Psychol 19(3):295-341

[48]

Raven J (1936) Mental tests used in genetic studies: the performances of related individuals in tests mainly educative and mainly reproductive. Master's thesis, University of London

[49]

Roethlisberger F (1939) Management and the worker. Harvard University Press, Cambridge

[50]

Rosenthal R, Jacobson L (1966) Teachers' expectancies: determinants of pupils' IQ gains. Psychol Rep 19(1):115-118

[51]

Sackman H, Erikson W, Grant E (1968) Exploratory experimental studies comparing online and offline programming performance. Commun ACM 11(1):3-11

Digital Library

[52]

Schlaug G (2001) The brain of musicians. A model for functional and structural adaptation. Ann N Y Acad Sci 930:281-299

[53]

Shadish W, Cook T, Campbell D (2002) Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company, Boston

[54]

Shaft T, Vessey I (1995) The relevance of application domain knowledge: the case of computer program comprehension. Inf Syst Res 6(3):286-299

Digital Library

[55]

Sharafi Z, Soh Z, Guéhéneuc YG, Antoniol G (2012) Women and men-different but equal: on the impact of identifier style on source code reading. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 27-36. IEEE CS

[56]

Sharif B, Maletic J (2009) An empirical study on the comprehension of stereotyped UML class diagram layouts. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 268-272. IEEE CS

[57]

Sharif B, Maletic J (2010) An eye tracking study on camel case and underscore identifier styles. In: Proc. Int'l Conf. Program Comprehension (ICPC), pp. 196-205. IEEE CS

[58]

Shneiderman B, Mayer R (1979) Syntactic/semantic interactions in programmer behavior: a model and experimental results. Int J Parallel Prog 8(3):219-238

[59]

Siegmund J (2012) Framework for measuring program comprehension. Ph.D. thesis, School of Computer Science, University of Magdeburg

[60]

Sjøberg D, Hannay J, Hansen O, Kampenes VB, Karahasanovic A, Liborg NK, Rekdal A (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31(9):733-753

Digital Library

[61]

Soloway E, Ehrlich K (1984) Empirical studies of programming knowledge. IEEE Trans Softw Eng 10(5):595-609

Digital Library

[62]

Standish T (1984) An essay on software reuse. IEEE Trans Softw Eng SE-10(5):494-497

[63]

Tiarks R (2011) What programmers really do: an observational study. In: Proc. Workshop Software Reengineering (WSR), pp. 36-37

[64]

Torchiano M (2004) Empirical assessment of UML static object diagrams. In: Proc. Int'l Workshop Program Comprehension (IWPC), pp. 226-230. IEEE CS

[65]

Vitharana P, Ramamurthy K (2003) Computer-mediated group support, anonymity, and the software inspection process: an empirical investigation. IEEE Trans Softw Eng 29(2):167-180

Digital Library

[66]

von Mayrhauser A, Vans M (1995) Program comprehension during software maintenance and evolution. Computer 28(8):44-55

Digital Library

[67]

von Mayrhauser A, Vans M, Howe A (1997) Program understanding behaviour during enhancement of large-scale software. J Softw Maint Res Pract 9(5):299-327

[68]

Wechsler D (1950) The measurement of adult intelligence, 3rd edn. American Psychological Association, Washington, DC

[69]

Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Boston

[70]

Wundt W (1874) Grundzüge der Physiologischen Psychologie. Engelmann, Leipzig

Cited By

Tempero EDenny PFinnie-Ansley JLuxton-Reilly AKirk DLeinonen JShakil ASheehan RTizard JTu YWuensche BBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)On the comprehensibility of functional decomposition: An empirical studyProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644432(214-224)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644432
Wyrich MApel S(2024)Evidence Tetris in the Pixelated World of Validity ThreatsProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648203(13-16)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648203
van de Laar PCorvino RMooij Avan Wezep HRosmalen R(2024)Custom static analysis to enhance insight into the usage of in-house librariesJournal of Systems and Software10.1016/j.jss.2024.112028212:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112028
Show More Cited By

Recommendations

On the Comprehension of Program Comprehension
Special Issue International Conference on Software Engineering (ICSE 2012) and Regular Papers

Research in program comprehension has evolved considerably over the past decades. However, only little is known about how developers practice program comprehension in their daily work. This article reports on qualitative and quantitative research to ...
Do Dynamic Object Process Graphs Support Program Understanding? - A Controlled Experiment.
ICPC '08: Proceedings of the 2008 The 16th IEEE International Conference on Program Comprehension

Using automatic program analysis techniques for extracting architectural information and its visualization is widely considered useful for program understanding. However, it has to be empirically validated if a given technique is beneficial in practice. ...
Comparing Trace Visualizations for Program Comprehension through Controlled Experiments
ICPC '15: Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension

For efficient and effective program comprehension, it is essential to provide software engineers with appropriate visualizations of the program's execution traces. Empirical studies, such as controlled experiments, are required to assess the ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 20, Issue 4

August 2015

314 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 August 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tempero EDenny PFinnie-Ansley JLuxton-Reilly AKirk DLeinonen JShakil ASheehan RTizard JTu YWuensche BBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)On the comprehensibility of functional decomposition: An empirical studyProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644432(214-224)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644432
Wyrich MApel S(2024)Evidence Tetris in the Pixelated World of Validity ThreatsProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648203(13-16)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648203
van de Laar PCorvino RMooij Avan Wezep HRosmalen R(2024)Custom static analysis to enhance insight into the usage of in-house librariesJournal of Systems and Software10.1016/j.jss.2024.112028212:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112028
Wyrich MBogner JWagner S(2023)40 Years of Designing Code Comprehension Experiments: A Systematic Mapping StudyACM Computing Surveys10.1145/362652256:4(1-42)Online publication date: 9-Nov-2023
https://dl.acm.org/doi/10.1145/3626522
Bouraffa AFuhrmann GMaalej WGrundy JPollock LPenta M(2023)Developers' Visuo-Spatial Mental Model and Program ComprehensionProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00163(1920-1932)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00163
Barón MWyrich MGraziotin DWagner SGrundy JPollock LPenta M(2023)Evidence Profiles for Validity Threats in Program Comprehension ExperimentsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00162(1907-1919)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00162
Stolp FGrundy J(2023)Assessing Cognitive Load in Software Development with Wearable SensorsProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00062(227-229)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00062
Krasniqi RDo H(2023)Towards semantically enhanced detection of emerging quality-related concerns in source codeSoftware Quality Journal10.1007/s11219-023-09614-831:3(865-915)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1007/s11219-023-09614-8
Al Madi N(2022)Namesake: A Checker of Lexical Similarity in Identifier NamesProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3560441(1-5)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3560441
Peitek NBergum ARekrut MMucke JNadig MParnin CSiegmund JApel SRoychoudhury ACadar CKim M(2022)Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking studyProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549084(120-131)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549084
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents