research-article

What works for whom, where, when, and why?: on the role of context in empirical software engineering

Authors:

Dag I.K. Sjøberg,

Daniela S. CruzesAuthors Info & Claims

ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement

Pages 19 - 28

https://doi.org/10.1145/2372251.2372256

Published: 19 September 2012 Publication History

Abstract

Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an in-dispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engineering research is mostly concerned with attempts to identify universal relationships that are independent of how work settings and other contexts interact with the processes important to software practice. The aim of this paper is to provide an overview of how context affects empirical research and how empirical software engineering research can be better 'contextualized' in order to provide a better understanding of what works for whom, where, when, and why. We exemplify the importance of context with examples from recent systematic reviews and offer recommendations on the way forward.

References

[1]

Arisholm, E. and Sjøberg, D.I.K. (2004) Evaluating the Effect of a Delegated versus Centralized Control Style on the Maintainability of Object-Oriented Software, IEEE Transactions on Software Engineering, 30(8): 521--534.

Digital Library

[2]

Arisholm, E., Gallis, H.E., Dybå, T. and Sjøberg, D.I.K. (2007) Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise, IEEE Transactions on Software Engineering, 33(2): 65--86.

Digital Library

[3]

Balijepally, V., Mahapatra, R., Nerur, S., Price, K.H. (2009) Are Two Heads Better Than One For Software Development? The Productivity Paradox of Pair Programming, MIS Quarterly, 33(1): 91--118.

[4]

Baron, R.M. and Kenny, D.A. (1993) The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations, Journal of Personality and Social Psychology, 51(6): 1173--1182.

[5]

Basili, V.R. and Rombach, D. (1988) The TAME Project: Towards Improvement-Oriented Software Environments, IEEE Transactions on Software Engineering, 14(6): 758--773.

Digital Library

[6]

Basili, V.R., Shull, F., and Lanubile, F. (1999) Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, 25(4): 456--473.

Digital Library

[7]

Bergersen, B.R. and Sjøberg, D.I.K. (2012) Evaluating Methods and Technologies in Software Engineering with Respect to Developers' Skill Level, Accepted to EASE'2012.

[8]

Bergersen, B.R., Hannay, J.E., Sjøberg, D.I.K., Dybå, T., and Karahasanović (2011) Inferring skill from tests of programming performance: Combining time and quality, Proc. ESEM'2011, IEEE Computer Society, pp. 305--314.

Digital Library

[9]

Brown, R. (2000) Group Processes: Dynamics within and between Groups, Second Ed., Blackwell.

[10]

Capretz, L.F. and Ahmed, F. (2010) Making Sense of Software Development and Personality Types, IT Professional, 12(1): 6--14.

Digital Library

[11]

Chin, E. (1994) Redefining "context" in research on writing, Written Communication, 11(4), 445--482.

[12]

Clarke, P. and O'Connor, R.V. (2012) The situational factors that affect the software development process: Towards a comprehensive reference framework, Information and Software Technology, 54(5): 433--447.

Digital Library

[13]

Cruzes, D.S. and Dybå, T. (2011) Research Synthesis in Software Engineering: A Tertiary Study, Information and Software Technology, 53(5): 440--455.

Digital Library

[14]

Cruzes, D.S. and Dybå, T. (2011) Recommended Steps for Thematic Synthesis in Software Engineering, Proc. ESEM'2011, IEEE Computer Society, pp. 275--284.

Digital Library

[15]

Dybå, T. (2000) Improvisation in Small Software Organizations, IEEE Software, 17(5): 82--87.

Digital Library

[16]

Dybå, T. (2003) Factors of Software Process Improvement Success in Small and Large Organizations: An Empirical Study in the Scandinavian Context, Proc. ESEC/FSE'2003, ACM press, pp. 148--157.

Digital Library

[17]

Dybå, T. (2003) A Dynamic Model of Software Engineering Knowledge Creation, in A. Aurum et al. (Eds.) Managing Software Engineering Knowledge, Springer, pp. 95--117.

[18]

Dybå, T., Arisholm, E., Sjøberg, D., Hannay, J., and Shull, F. (2007) Are Two Heads Better than One? On the Effectiveness of Pair-Programming, IEEE Software, 24(6): 12--15.

Digital Library

[19]

Dybå, T., Kitchenham, B.A., and Jørgensen, M. (2005) Evidence-based Software Engineering for Practitioners, IEEE Software, 22(1): 58--65.

Digital Library

[20]

Dybå, T., Moe, N.B., and Arisholm, E. (2005) Measuring Software Methodology Usage: Challenges of Conceptualization and Operationalization, Proc. ISESE'2005, pp. 447--457

[21]

Dybå, T., Prikladnicki, R., Rönkkö, K., Seaman, C., and Sillito, J. (2011) Qualitative Research in Software Engineering, Empirical Software Engineering, 16(4): 425--429.

Digital Library

[22]

Ericsson, K.A. and Charness, N. (1994) Expert Performance: Its Structure and Acquisition, American Psychologist, 49(8): 725--747.

[23]

Fenton, N., Pfleeger, S.L. and Glass, R.L. (1994) Science and Substance: A Challenge to Software Engineers, IEEE Software, 11(4): 86--95.

Digital Library

[24]

Goodwin, C. and Duranti, A. (1992) Rethinking context: Language as an interactive phenomenon, Cambridge Univ. Press.

[25]

Griffin, M. (2007) Specifying organizational contexts: Systematic links between contexts and processes in organizational behavior, Journal of Organizational Behavior, 28: 859--863.

[26]

Hannay, J., Dybå, T., Arisholm, E., and Sjøberg, D. (2009) The Effectiveness of Pair Programming: A Meta-Analysis, Information and Software Technology, 51(7): 1110--1122.

Digital Library

[27]

Hedges, L.V. (1987) How Hard Is Hard Science, How Soft Is Soft Science? The Empirical Cumulativeness of Research, American Psychologist, 42(2): 443--455.

[28]

Höst, M., Regnell, B., and Wohlin, C. (2000) Using Students as Subjects: A Comparative Study of Students and Professionals in Lead-Time Impact Assessment, Empirical Software Engineering, 5(3): 201--214.

Digital Library

[29]

Johns, G. (1991) Substantive and methodological constraints on behavior and attitudes in organizational research. Organizational Behavior and Human Decision Processes, 49: 80--104.

[30]

Johns, G. (2006) The Essential Impact of Context on Organizational Behavior, Academy of Management Review, 31(2): 386--408.

[31]

Karau, S.J. and Williams, K.D. (1993) Social loafing: a meta-analytic review and theoretical (integration, Journal of Personality and Social Psychology, 65(4): 681--706.

[32]

Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K. and Rosenberg, J. (2002) Preliminary Guidelines for Empirical Research in Software Engineering, IEEE Transactions on Software Engineering, 28(8): 721--734.

Digital Library

[33]

Menzies, T., Butcher, A., Marcus, A., Zimmermann, T., and Cok, D. (2011) Local vs. Global Models for Effort Estimation and Defect Prediction, Proc. ASE'2011, pp. 343--351.

Digital Library

[34]

Michailova, S. (2011) Contextualizing in International Business research: Why do we need more of it and how can we be better at it? Scandinavian Journal of Management, 27: 129--139.

[35]

Moe, N.B., Dingsøyr, T. and Dybå, T. (2009) Overcoming Barriers to Self-management in Software Teams, IEEE Software, 26(6): 20--26.

Digital Library

[36]

Mowday, R. and Sutton, R. (1993) Organizational behavior: Linking individuals and groups to organizational contexts, Annual Review of Psychology, 44: 195--229.

[37]

Petersen K. and Wohlin, C. (2009) Context in Industrial Software Engineering Research, Proc. ESEM'2009, pp. 401--404.

Digital Library

[38]

Rousseau, D.M. and Fried, Y. (2001) Location, location, location: contextualizing organizational research, Journal of Organizational Behavior, 22: 1--13.

[39]

Shull, F. (2012) I Believe! IEEE Software, 29(1): 4--7.

Digital Library

[40]

Sjøberg, D, Dybå, T., and Jørgensen, M. (2007) The Future of Empirical Methods in Software Engineering Research, Proc. FOSE'2007, pp. 358--378.

Digital Library

[41]

Sjøberg, D., Dybå, T., Anda, B., and Hannay, J. (2008) Building Theories in Software Engineering, in F. Shull, J. Singer, and D. Sjøberg (Eds.) Advanced Topics in Empirical Software Engineering, Springer, pp. 312--336.

[42]

Sjøberg, D.I.K., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Koren, E.F. and Vokac M. (2002) Conducting Realistic Experiments in Software Engineering, Proc. ISESE'2002, pp. 17--26.

Digital Library

[43]

Sjøberg, D.I.K., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., and Vokac M. (2003) Challenges and Recommendation when Increasing the Realism of Controlled Software Engineering Experiments, in R. Conradi & A.I. Wang (Eds.) Empirical Methods and Studies in Software Engineering - Experiences from ESERNET, Springer, LNCS 2765, pp. 24--38.

[44]

Sjøberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanović, A., Liborg, N.-K. and Rekdal, A.C. (2005) A Survey of Controlled Experiments in Software Engineering, IEEE Transactions on Software Engineering, 31(9): 733--753.

Digital Library

[45]

Trist, E. (1981) The Evolution of Socio-Technical Systems: A Conceptual Framework and an Action Research Program, Occasional papers No. 2, Ontario Quality of Working Life Center.

[46]

Turhan, B., Layman, L., Diep, M., Shull, F. and Erdogmus, H. (2010) How Effective is Test Driven Development?, in G.Wilson & A. Orham (Eds.), Making Software: What Really Works, and Why We Believe It, O'Reilly Press, pp. 207--219.

[47]

Welter, F. (2010) Contextualizing Entrepreneurship: Conceptual Challenges and Ways Forward, Entrepreneurship Theory and Practice, 35(1): 165--184.

[48]

Whetten (2009) An Examination of the Interface between Context and Theory Applied to the Study of Chinese Organizations, Management and Organization Review, 5(1): 29--55.

[49]

Whetten, D.A. (1989) What Constitutes a Theoretical Contribution, Academy of Management Review, 14(4): 490--495.

[50]

Xu, P. and Ramesh, B. (2007) Software process tailoring: an empirical investigation, Journal of Management Information Systems, 24(2): 293--328.

Digital Library

Cited By

Heldal R(2024)Is generalisation hindering the adoption of your findings?Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686694(348-358)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686694
Neumann MSchmid KBaumann L(2024)What You Use is What You Get: Unforced Errors in Studying Cultural Aspects in Agile Software DevelopmentProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661229(405-410)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661229
Vishnubhotla SMendes E(2024)Exploring the relation between personality traits and agile team climateJournal of Systems and Software10.1016/j.jss.2023.111937210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111937
Show More Cited By

Index Terms

What works for whom, where, when, and why?: on the role of context in empirical software engineering
1. Software and its engineering

Recommendations

The turn to practice in HCI: towards a research agenda
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

This paper argues that a new paradigm for HCI research, which we label the 'practice' perspective, has been emerging in recent years. This stands in contrast to the prevailing mainstream HCI paradigm, which we term the 'interaction' perspective. The '...
Engaging the net generation with evidence-based software engineering through a community-driven web database

Software engineering faculty face the challenge of educating future researchers and industry practitioners regarding the generation of empirical software engineering studies and their use in evidence-based software engineering. In order to engage the ...
Ten years with evidence-based software engineering. What is it? Has it had any impact? What's next?
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

An evidence-based software engineer is one who is able to: 1) Formulate a question, related to a decision or judgment, so that it can be answered by the use of evidence, 2) Collect, critically evaluate and summarise relevant evidence from research, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement

September 2012

338 pages

ISBN:9781450310567

DOI:10.1145/2372251

General Chair:
Per Runeson
Lund University, Sweden
,
Program Chairs:
Martin Höst
Lund University, Sweden
,
Emilia Mendes
Zayed University, United Arab Emirates
,
Anneliese Andrews
University of Denver, USA
,
Rachel Harrison
Oxford Brookes University, UK

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEM '12

Sponsor:

SIGSOFT

ESEM '12: 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 19 - 20, 2012

Lund, Sweden

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

88
Total Citations
View Citations
997
Total Downloads

Downloads (Last 12 months)111
Downloads (Last 6 weeks)11

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Heldal R(2024)Is generalisation hindering the adoption of your findings?Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686694(348-358)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686694
Neumann MSchmid KBaumann L(2024)What You Use is What You Get: Unforced Errors in Studying Cultural Aspects in Agile Software DevelopmentProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661229(405-410)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661229
Vishnubhotla SMendes E(2024)Exploring the relation between personality traits and agile team climateJournal of Systems and Software10.1016/j.jss.2023.111937210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111937
Papatheocharous EWohlin CBadampudi DCarlson JWnuk K(2024)Context factors perceived important when looking for similar experiences in decision‐making for software components: An interview studyJournal of Software: Evolution and Process10.1002/smr.266836:9Online publication date: 11-Apr-2024
https://doi.org/10.1002/smr.2668
Pretorius CRazavian MEling KLangerak F(2024)When rationality meets intuition: A research agenda for software design decision‐makingJournal of Software: Evolution and Process10.1002/smr.266436:9Online publication date: 31-Mar-2024
https://doi.org/10.1002/smr.2664
Ghaleb THassan SZou Y(2023)Studying the Interplay Between the Durations and Breakages of Continuous Integration BuildsIEEE Transactions on Software Engineering10.1109/TSE.2022.322216049:4(2476-2497)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3222160
Börstler JAli NSvensson MPetersen K(2023)Investigating acceptance behavior in software engineering—Theoretical perspectivesJournal of Systems and Software10.1016/j.jss.2022.111592198(111592)Online publication date: Apr-2023
https://doi.org/10.1016/j.jss.2022.111592
Razavian MPaech BTang A(2023)The vision of on-demand architectural knowledge systems as a decision-making companionJournal of Systems and Software10.1016/j.jss.2022.111560198:COnline publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111560
Frattini JMontgomery LFischbach JMendez DFucci DUnterkalmsteiner M(2023)Requirements quality research: a harmonized theory, evaluation, and roadmapRequirements Engineering10.1007/s00766-023-00405-y28:4(507-520)Online publication date: 12-Aug-2023
https://doi.org/10.1007/s00766-023-00405-y
Rainer AWohlin C(2022)Recruiting credible participants for field studies in software engineering researchInformation and Software Technology10.1016/j.infsof.2022.107002151:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.infsof.2022.107002
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents