research-article

A characterization study of testing contributors and their contributions in open source projects.

Authors:

Hugo Henrique Fumero de Souza,

Igor Steinmacher,

Reginaldo RéAuthors Info & Claims

SBES '22: Proceedings of the XXXVI Brazilian Symposium on Software Engineering

Pages 95 - 105

https://doi.org/10.1145/3555228.3555244

Published: 05 October 2022 Publication History

Abstract

Even though open source projects have some different characteristics from projects in the industry, the commitment of maintainers and contributors to achieve a high level of software quality is constant. Therefore, tests are among the main practices of the communities. Thus, motivating contributors to write new tests and maintain regression tests during testing activities is essential for the project’s health. The objective of our work is to characterize testers and their contributions to open source projects as part of a broad study about testers’ motivation. Thus, we conducted a study with 3,936 repositories and 7 different and important programming languages (C, C++, C#, Java, Javascript, Python, and Ruby), analyzing a total of 4,409,142 contributions to classify contributing members and their contributions. Our results show that test-only contributors exist, regardless of programming language or project. We conclude that, despite the unfavorable scenario, there are contributors who feel motivated and dedicate their time and effort to contribute to new tests or to the evolution of existing tests.

References

[1]

Tamer Abdou, Peter Grogono, and Pankaj Kamthan. 2012. A Conceptual Framework for Open Source Software Test Process, In 2012 IEEE 36th Annual Computer Software and Applications Conference Workshops. Proceedings - International Computer Software and Applications Conference, 458–463.

[2]

Mark Aberdour. 2007. Achieving Quality in Open-Source Software. IEEE Software 24, 1 (2007), 58–64.

Digital Library

[3]

Adam Alami, Marisa Leavitt Cohn, and Andrzej Waisowski. 2020. How Do FOSS Communities Decide to Accept Pull Requests?. In Proceedings of the Evaluation and Assessment in Software Engineering (Trondheim, Norway) (EASE ’20). Association for Computing Machinery, New York, NY, USA, 220–229.

Digital Library

[4]

Adam Alami, Yvonne Dittrich, and Andrzej Wasowski. 2018. Influencers of Quality Assurance in an Open Source Community. In Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering (Gothenburg, Sweden) (CHASE ’18). Association for Computing Machinery, New York, NY, USA, 61–68.

Digital Library

[5]

Ann Barcomb, Klaas-Jan Stol, Brian Fitzgerald, and Dirk Riehle. 2022. Managing Episodic Volunteers in Free/Libre/Open Source Software Communities. IEEE Transactions on Software Engineering 48, 1 (2022), 260–277.

Digital Library

[6]

Adina Barham. 2013. The Emergence of Quality Assurance Practices in Free/Libre Open Source Software: A Case Study. In Open Source Software: Quality Verification, Etiel Petrinja, Giancarlo Succi, Nabil El Ioini, and Alberto Sillitti (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 271–276.

[7]

Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman. 2015. When, how, and why developers (do not) test in their IDEs. 2015 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Proceedings(2015), 179–190.

Digital Library

[8]

Marcus Vinicius Bertoncello, Gustavo Pinto, Igor Scaliante Wiese, and Igor Steinmacher. 2020. Pull Requests or Commits? Which Method Should We Use to Study Contributors’ Behavior?. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 592–601.

[9]

Andrea Capiluppi, Patricia Lago, and Maurizio Morisio. 2003. Characteristics of open source projects. 317– 327.

[10]

Andrea Capiluppi and Martin Michlmayr. 2007. From the Cathedral to the Bazaar: An Empirical Study of the Lifecycle of Volunteer Community Projects. In Open Source Development, Adoption and Innovation, Joseph Feller, Brian Fitzgerald, Walt Scacchi, and Alberto Sillitti (Eds.). Springer US, Boston, MA, 31–44.

[11]

Luiz Fernando Capretz, Daniel Varona, and Arif Raza. 2015. Influence of personality types in software tasks choices. Computers in Human Behavior 52 (2015), 373–378.

Digital Library

[12]

Luiz Fernando Capretz, Pradeep Waychal, Jingdong Jia, Daniel Varona, and Yadira Lizama. 2019. Studies on the Software Testing Profession. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 262–263.

[13]

Jacob Cohen. 1988. Statistical power analysis for the behavioral sciences (2nd edition ed.). Routledge. 567 pages.

[14]

Anca Deak. 2014. A comparative study of testers’ motivation in traditional and agile software development. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8892 (2014), 1–16.

[15]

Anca Deak, Tor Stålhane, and Guttorm Sindre. 2016. Challenges and strategies for motivating software testing personnel. Information and Software Technology 73 (2016), 1–15.

Digital Library

[16]

Anca Deak, Tor Stålhane, and Daniela Cruzes. 2013. Factors Influencing the Choice of a Career in Software Testing among Norwegian Students. IASTED Multiconferences - Proceedings of the IASTED International Conference on Software Engineering, SE 2013 (03 2013).

[17]

Vieri Del Bianco, Luigi Lavazza, Sandro Morasca, Davide Taibi, and Davide Tosi. 2010. An investigation of the users’ perception of OSS quality. IFIP Advances in Information and Communication Technology 319 AICT(2010), 15–28.

[18]

Matthieu Foucault, Marc Palyart, Xavier Blanc, Gail C. Murphy, and Jean-Rémy Falleri. 2015. Impact of Developer Turnover on Quality in Open-Source Software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 829–841.

Digital Library

[19]

A. César C. França and Fabio Q.B. Da Silva. 2009. An empirical study on software engineers motivational factors. 2009 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM 2009 (2009), 405–409.

[20]

César França, Fabio QB Da Silva, and Helen Sharp. 2018. Motivation and satisfaction of software engineers. IEEE Transactions on Software Engineering 46, 2 (2018), 118–140.

[21]

César França, Helen Sharp, and Fabio Silva. 2014. Motivated software engineers are engaged and focused, while satisfied ones are happy. International Symposium on Empirical Software Engineering and Measurement (09 2014).

Digital Library

[22]

Felipe Fronchetti, Igor Wiese, Gustavo Pinto, and Igor Steinmacher. 2019. What Attracts Newcomers to Onboard on OSS Projects? TL;DR: Popularity. In Open Source Systems, Francis Bordeleau, Alberto Sillitti, Paulo Meirelles, and Valentina Lenarduzzi (Eds.). Springer International Publishing, Cham, 91–103.

[23]

Marco Gerosa, Igor Wiese, Bianca Trinkenreich, Georg Link, Gregorio Robles, Christoph Treude, Igor Steinmacher, and Anita Sarma. 2021. The Shifting Sands of Motivation: Revisiting What Drives Contributors in Open Source. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 1046–1058.

[24]

Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An Exploratory Study of the Pull-Based Software Development Model. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 345–355.

Digital Library

[25]

A. Hars and S. Ou. 2001. Working for free? - Motivations of participating in open source projects. Proceedings of the Hawaii International Conference on System Sciences 00, c(2001), 163.

[26]

Frederick Herzberg, Bernard Mausner, and Barbara Snyderman. 1959. The motivation to work, 2nd ed.John Wiley, Oxford, England. xv, 157–xv, 157 pages.

[27]

Samoladas I, Gousios G, Spinellis D, and Stamelos I.2008. The SQO-OSS quality model: measurement based open source software evaluation. Open source development, communities and quality. The International Federation for Information Processing IFIP 275 (2008), 237–48.

[28]

Tanjila Kanij, Robert Merkel, and John Grundy. 2015. An Empirical Investigation of Personality Traits of Software Testers. In Proceedings of the Eighth International Workshop on Cooperative and Human Aspects of Software Engineering (Florence, Italy) (CHASE ’15). IEEE Press, 1–7.

Digital Library

[29]

Atieh Khanjani and Riza Sulaiman. 2011. The process of quality assurance under open source software development. In 2011 IEEE Symposium on Computers Informatics. 548–552.

[30]

Foutse Khomh, Bram Adams, Tejinder Dhaliwal, and Ying Zou. 2015. Understanding the Impact of Rapid Releases on Software Quality. Empirical Softw. Engg. 20, 2 (April 2015), 336–373.

Digital Library

[31]

Pavneet Singh Kochhar, Tegawendé F. Bissyandé, David Lo, and Lingxiao Jiang. 2013. Adoption of software testing in open source projects - A preliminary study on 50,000 projects. Proceedings of the European Conference on Software Maintenance and Reengineering, CSMR (2013), 353–356. https://doi.org/10.1109/CSMR.2013.48

Digital Library

[32]

Amanda Lee, Jeffrey C. Carver, and Amiangshu Bosu. 2017. Understanding the Impressions, Motivations, and Barriers of One Time Code Contributors to FLOSS Projects: A Survey. Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering, ICSE 2017(2017), 187–197.

Digital Library

[33]

Sandro Morasca, Davide Taibi, and Davide Tosi. 2011. OSS-TMM: Guidelines for Improving the Testing Process of Open Source Software. Int. J. Open Source Softw. Process. 3, 2 (April 2011), 1–22.

Digital Library

[34]

Anh Nguyen-Duc, Daniela S. Cruzes, and Reidar Conradi. 2015. The impact of global dispersion on coordination, team performance and software quality-A systematic literature review. Information and Software Technology 57, 1 (2015), 277–294.

[35]

Raphael Pham, Leif Singer, Olga Liskin, Fernando Figueira Filho, and Kurt Schneider. 2013. Creating a shared understanding of testing culture on a social coding site. In 2013 35th International Conference on Software Engineering (ICSE). 112–121.

[36]

Gustavo Pinto, Igor Steinmacher, and Marco Aurélio Gerosa. 2016. More Common Than You Think: An In-depth Study of Casual Contributors. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 112–123.

[37]

Gustavo Pinto, Igor Steinmacher, and Marco Aurélio Gerosa. 2016. More Common Than You Think: An In-depth Study of Casual Contributors. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 112–123.

[38]

Muh Riansyah. 2015. What motivate software tester ? A Systematic Literature Review. November (2015), 1–5.

[39]

Peter Rigby, Brendan Cleary, Frederic Painchaud, Margaret-Anne Storey, and Daniel German. 2012. Contemporary Peer Review in Action: Lessons from Open Source Development. IEEE Softw. 29, 6 (Nov. 2012), 56–61.

Digital Library

[40]

Richard M. Ryan and Edward L. Deci. 2000. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary Educational Psychology 25, 1 (2000), 54–67.

[41]

Ronnie Edson De Souza Santos, Cleyton Vanut Cordeiro De Magalhaes, Jorge Da Silva Correia-Neto, Fabio Queda Bueno Da Silva, Luiz Fernando Capretz, and Rodrigo Souza. 2017. Would You Like to Motivate Software Testers? Ask Them How. International Symposium on Empirical Software Engineering and Measurement 2017-Novem(2017), 95–104.

[42]

Bianca Shibuya and Tetsuo Tamai. 2009. Understanding the process of participating in open source communities. Proceedings of the 2009 ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development, FLOSS 2009 (2009), 1–6.

Digital Library

[43]

Igor Steinmacher, Tayana Uchôa Conte, Marco Aurélio Gerosa, and David F. Redmiles. 2015. Social barriers faced by newcomers placing their first contribution in open source software projects. CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social ComputingOctober 2014 (2015), 1379–1392.

Digital Library

[44]

Valerio Terragni, Pasquale Salza, and Mauro Pezzè. 2020. Measuring Software Testability Modulo Test Quality. In Proceedings of the 28th International Conference on Program Comprehension (Seoul, Republic of Korea) (ICPC ’20). Association for Computing Machinery, New York, NY, USA, 241–251.

Digital Library

[45]

Christopher Thompson. 2017. Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software. Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering(2017), 83–92.

Digital Library

[46]

Georg Von Krogh, Stefan Haefliger, Sebastian Spaeth, and Martin W. Wallin. 2012. Carrots and Rainbows: Motivation and Social Practice in Open Source Software Development. MIS Q. 36, 2 (June 2012), 649–676.

Digital Library

[47]

Pradeep Kashinath Waychal and Luiz Fernando Capretz. 2016. Why a testing career is not the first choice of engineers. ASEE Annual Conference and Exposition, Conference Proceedings 2016-June, June(2016).

Cited By

Rao NJain KAlon UGoues CHellendoorn V(2023)CAT-LM Training Language Models on Aligned Code And Tests2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00193(409-420)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00193

Index Terms

A characterization study of testing contributors and their contributions in open source projects.
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
2. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software configuration management and version control systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Students' Engagement in Open Source Projects: An Analysis of Google Summer of Code
SBES '17: Proceedings of the XXXI Brazilian Symposium on Software Engineering

Several open source software (OSS) communities promote and participate in initiatives such as summers of code to foster contributions and attract new developers. However, little is known about how successful these initiatives are. As a case study, we ...
Conversational Bot for Newcomers Onboarding to Open Source Projects
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

This paper targets the problems newcomers face when onboarding to open source projects and the low retention rate of newcomers. Open source software projects are becoming increasingly more popular. Many major companies have started building open source ...
Overcoming Social Barriers When Contributing to Open Source Software Projects

An influx of newcomers is critical to the survival, long-term success, and continuity of many Open Source Software (OSS) community-based projects. However, newcomers face many barriers when making their first contribution, leading in many cases to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBES '22: Proceedings of the XXXVI Brazilian Symposium on Software Engineering

October 2022

457 pages

ISBN:9781450397353

DOI:10.1145/3555228

Editors:
Marcelo Maia
Universidade Federal de Uberlândia (UFU), Brazil
,
Fabiano Dorça
Universidade Federal de Uberlândia (UFU), Brazil
,
Rafael Araújo
Universidade Federal de Uberlândia (UFU), Brazil
,
Christina von Flach
Universidade Federal da Bahia (UFBA), Brazil
,
Elisa Yumi Nakagawa
Universidade de São Paulo (ICMC-USP), Brazil
,
Edna Dias Canedo
Universidade de Brasília (UnB), Brazil

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

SBES 2022

SBES 2022: XXXVI Brazilian Symposium on Software Engineering

October 5 - 7, 2022

Virtual Event, Brazil

Acceptance Rates

Overall Acceptance Rate 147 of 427 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
74
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rao NJain KAlon UGoues CHellendoorn V(2023)CAT-LM Training Language Models on Aligned Code And Tests2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00193(409-420)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00193

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten