short-paper

Standards of validity and the validity of standards in behavioral software engineering research: the perspective of psychological test theory

Author:

Lucas GrenAuthors Info & Claims

ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 55, Pages 1 - 4

https://doi.org/10.1145/3239235.3267437

Published: 11 October 2018 Publication History

Abstract

Background. There are some publications in software engineering research that aim at guiding researchers in assessing validity threats to their studies. Still, many researchers fail to address many aspects of validity that are essential to quantitative research on human factors.

Goal. This paper has the goal of triggering a change of mindset in what types of studies are the most valuable to the behavioral software engineering field, and also provide more details of what construct validity is.

Method. The approach is based on psychological test theory and draws upon methods used in psychology in relation to construct validity.

Results. In this paper, I suggest a different approach to validity threats than what is commonplace in behavioral software engineering research.

Conclusions. While this paper focuses on behavioral software engineering, I believe other types of software engineering research might also benefit from an increased focus on construct validity.

References

[1]

American Educational Research Association. 2014. Standards for educational and psychological testing. American Educational Research Association, Washington, DC.

[2]

Lee Cronbach. 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16, 3 (1951), 297--334.

[3]

Robert Feldt and Ana Magazinius. 2010. Validity Threats in Empirical Software Engineering Research - An Initial Survey. In International Conference on Software Engineering and Knowledge Engineering (SEKE). 374--379.

[4]

Scott Freeman, Sarah L. Eddy, Miles McDonough, Michelle K. Smith, Nnadozie Okoroafor, Hannah Jordt, and Mary Pat Wenderoth. 2014. Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences 111, 23 (2014), 8410--8415.

[5]

Thomas Gilovich. 1983. Biased evaluation and persistence in gambling. Journal of personality and social psychology 44, 6 (1983), 1110.

[6]

Zvi Ginossar and Yaacov Trope. 1987. Problem solving in judgment under uncertainty. Journal of Personality and social Psychology 52, 3 (1987), 464.

[7]

Lucas Gren. 2017. Psychological group processes when building agile software development teams. Ph.D. Dissertation. The University of Gothenburg.

[8]

Andreas Jedlitschka and Dietmar Pfahl. 2005. Reporting guidelines for controlled experiments in software engineering. In International Symposium on Empirical Software Engineering. IEEE, 95--104.

[9]

Magne Jørgensen, Tore Dybå, Knut Liestøl, and Dag IK Sjøberg. 2016. Incorrect results in software engineering experiments: How to improve research practices. Journal of Systems and Software 116 (2016), 133--145.

Digital Library

[10]

Magne Jørgensen and Efi Papatheocharous. 2015. Believing is seeing: Confirmation bias studies in software engineering. In 41st Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 92--95.

Digital Library

[11]

Daniel Kahneman. 2011. Thinking, fast and slow. Farrar, Straus and Giroux, New York.

[12]

G Frederic Kuder and Marion W Richardson. 1937. The theory of the estimation of test reliability. Psychometrika 2, 3 (1937), 151--160.

[13]

Ziva Kunda. 1990. The case for motivated reasoning. Psychological bulletin 108, 3 (1990), 480.

[14]

Samuel Messick. 1995. Standards of validity and the validity of standards in performance assessment. Educational measurement: Issues and practice 14, 4 (1995), 5--8.

[15]

Samuel Messick. 1995. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American psychologist 50, 9 (1995), 741.

[16]

Raymond S Nickerson. 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2 (1998), 175.

[17]

Tinca JC Polderman, Beben Benyamin, Christiaan A De Leeuw, Patrick F Sullivan, Arjen Van Bochoven, Peter M Visscher, and Danielle Posthuma. 2015. Meta-analysis of the heritability of human traits based on fifty years of twin studies. Nature genetics 47, 7 (2015), 702--709.

[18]

Tom Pyszczynski and Jeff Greenberg. 1987. Toward an integration of cognitive and motivational perspectives on social inference: A biased hypothesis-testing model. Advances in experimental social psychology 20 (1987), 297--340.

[19]

P. Runeson and M. Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14, 2 (2009), 131--164.

Digital Library

[20]

Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer, Berlin.

Cited By

Curci A(2024)A Comprehensive Framework Proposal to Design Symbiotic AI SystemsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661219(460-465)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661219
Dawood JGren L(2024)Prototypical Leadership in Agile Software DevelopmentProduct-Focused Software Process Improvement10.1007/978-3-031-78386-9_13(189-203)Online publication date: 27-Nov-2024
https://doi.org/10.1007/978-3-031-78386-9_13
Liu XChen ZQian YZhong CHuang HLi SShao D(2024)Towards a security‐optimized approach for the microservice‐oriented decompositionJournal of Software: Evolution and Process10.1002/smr.2670Online publication date: 27-Jun-2024
https://doi.org/10.1002/smr.2670
Show More Cited By

Index Terms

Standards of validity and the validity of standards in behavioral software engineering research: the perspective of psychological test theory
1. General and reference
  1. Cross-computing tools and techniques

Recommendations

Construct Validity in Software Engineering Research and Software Metrics
EASE '18: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018

Construct validity is essentially the degree to which our scales, metrics and instruments actually measure the properties they are supposed to measure. Although construct validity is widely considered an important quality criterion for most empirical ...
Validity concerns in software engineering research
FoSER '10: Proceedings of the FSE/SDP workshop on Future of software engineering research

Empirical studies that use software repository artifacts have become popular in the last decade due to the ready availability of open source project archives. In this paper, we survey empirical studies in the last three years of ICSE and FSE proceedings,...
Software Engineering Project Standards

Software Engineering Project Standards (SEPS) and their importance are presented in this paper by looking at standards in general, then progressively narrowing the view to software standards, to software engineering standards, and finally to SEPS. After ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

October 2018

487 pages

ISBN:9781450358231

DOI:10.1145/3239235

General Chair:
Markku Oivo
University of Oulu, Finland
,
Program Chairs:
Daniel Méndez
Technical University of Munich, Germany
,
Audris Mockus
University of Tennessee

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ESEM '18

Sponsor:

SIGSOFT

ESEM '18: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

October 11 - 12, 2018

Oulu, Finland

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
215
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Curci A(2024)A Comprehensive Framework Proposal to Design Symbiotic AI SystemsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661219(460-465)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661219
Dawood JGren L(2024)Prototypical Leadership in Agile Software DevelopmentProduct-Focused Software Process Improvement10.1007/978-3-031-78386-9_13(189-203)Online publication date: 27-Nov-2024
https://doi.org/10.1007/978-3-031-78386-9_13
Liu XChen ZQian YZhong CHuang HLi SShao D(2024)Towards a security‐optimized approach for the microservice‐oriented decompositionJournal of Software: Evolution and Process10.1002/smr.2670Online publication date: 27-Jun-2024
https://doi.org/10.1002/smr.2670
Lenberg PFeldt RGren LWallgren Tengberg LTidefors IGraziotin D(2024)Qualitative software engineering researchJournal of Software: Evolution and Process10.1002/smr.260736:6Online publication date: 5-Jun-2024
https://dl.acm.org/doi/10.1002/smr.2607
Graf-Vlachy LWagner S(2023)The Type to Take Out a Loan? A Study of Developer Personality and Technical Debt2023 ACM/IEEE International Conference on Technical Debt (TechDebt)10.1109/TechDebt59074.2023.00010(27-36)Online publication date: May-2023
https://doi.org/10.1109/TechDebt59074.2023.00010
Zhang XYu YGousios GRastogi A(2023)Pull Request Decisions Explained: An Empirical OverviewIEEE Transactions on Software Engineering10.1109/TSE.2022.316505649:2(849-871)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TSE.2022.3165056
Gren LShepperd MBaldassarre MPrechelt LRodríguez-Pérez G(2022)Problem reports and team maturity in Agile automotive software developmentProceedings of the 15th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3528579.3529173(41-45)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3528579.3529173
Russo DStol K(2022)Gender Differences in Personality Traits of Software EngineersIEEE Transactions on Software Engineering10.1109/TSE.2020.300341348:3(819-834)Online publication date: 1-Mar-2022
https://doi.org/10.1109/TSE.2020.3003413
Penzenstadler BTorkar RMartinez Montes C(2022)Take a deep breath: Benefits of neuroplasticity practices for software developers and computer workers in a family of experimentsEmpirical Software Engineering10.1007/s10664-022-10148-z27:4Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s10664-022-10148-z
Russo DMasegosa AStol K(2022)From anecdote to evidence: the relationship between personality and need for cognition of developersEmpirical Software Engineering10.1007/s10664-021-10106-127:3Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s10664-021-10106-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents