Cited By
View all- Murali ASahu GThangarajah KZimmerman BRodríguez-Pérez GNagappan M(2024)Diversity in issue assignment: humans vs botsEmpirical Software Engineering10.1007/s10664-023-10424-629:2Online publication date: 9-Jan-2024
Systemic Gender Inequities in Who Reviews Code
Authors: 
Emerson Murphy-Hill, Jillian Dicker, Amber Horvath, Maggie Morrow Hodges, Carolyn D. Egelman, Laurie R. Weingart, Ciera Jaspan, Collin Green, Nina ChenAuthors Info & Claims
Emerson Murphy-Hill, Jillian Dicker, Amber Horvath, Maggie Morrow Hodges, Carolyn D. Egelman, Laurie R. Weingart, Ciera Jaspan, Collin Green, Nina ChenAuthors Info & ClaimsArticle No.: 94, Pages 1 - 59
Abstract
Code review is an essential task for modern software engineers, where the author of a code change assigns other engineers the task of providing feedback on the author's code. In this paper, we investigate the task of code review through the lens of equity, the proposition that engineers should share reviewing responsibilities fairly. Through this lens, we quantitatively examine gender inequities in code review load at Google. We found that, on average, women perform about 25% fewer reviews than men, an inequity with multiple systemic antecedents, including authors' tendency to choose men as reviewers, a recommender system's amplification of human biases, and gender differences in how reviewer credentials are assigned and earned. Although substantial work remains to close the review load gap, we show how one small change has begun to do so.
References
[1]
Wisam Haitham Abbood Al-Zubaidi, Patanamon Thongtanunam, Hoa Khanh Dam, Chakkrit Tantithamthavorn, and Aditya Ghose. 2020. Workload-aware reviewer recommendation using a multi-objective search-based approach. In Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering. 21--30.
[2]
Sumit Asthana, Rahul Kumar, Ranjita Bhagwan, Christian Bird, Chetan Bansal, Chandra Maddila, Sonu Mehta, and B Ashok. 2019. WhoDo: automating reviewer suggestions at scale. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 937--945.
[3]
Linda Babcock, Brenda Peyser, Lise Vesterlund, and Laurie Weingart. 2022. The No Club: Putting a Stop to Women's Dead End Work. Simon & Schuster, New York, New York.
[4]
Linda Babcock, Maria P Recalde, Lise Vesterlund, and Laurie Weingart. 2017. Gender differences in accepting and receiving requests for tasks with low promotability. American Economic Review, Vol. 107, 3 (2017), 714--47.
[5]
Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In ICSE. IEEE Press, San Francisco, California, 712--721.
[6]
V. Balachandran. 2013. Fix-it: An extensible code auto-fix component in Review Bot. In Source Code Analysis and Manipulation (SCAM), 2013 IEEE 13th International Working Conference on. 167--172. https://doi.org/10.1109/SCAM.2013.6648198
[7]
Shaowen Bardzell and Jeffrey Bardzell. 2011. Towards a feminist HCI methodology: social science, feminism, and HCI. In Proceedings of the SIGCHI conference on human factors in computing systems. 675--684.
[8]
Olga Baysal, Oleksii Kononenko, Reid Holmes, and Michael W Godfrey. 2013. The influence of non-technical factors on code review. In WCRE.
[9]
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the royal statistical society. Series B (Methodological) (1995), 289--300.
[10]
Sylvia Beyer. 2014. Why are women underrepresented in Computer Science? Gender differences in stereotypes, self-efficacy, values, and interests and predictors of future CS course-taking and grades. Computer Science Education, Vol. 24, 2--3 (2014), 153--192.
[11]
Katerina Bezrukova, Chester S Spell, Jamie L Perry, and Karen A Jehn. 2016. A meta-analytical integration of over 40 years of research on diversity training evaluation. Psychological Bulletin, Vol. 142, 11 (2016), 1227.
[12]
Amiangshu Bosu, Jeffrey C Carver, Christian Bird, Jonathan Orbeck, and Christopher Chockley. 2016. Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Transactions on Software Engineering, Vol. 43, 1 (2016), 56--75.
[13]
Amiangshu Bosu and Kazi Zakia Sultana. 2019. Diversity and inclusion in open source software (OSS) projects: Where do we stand?. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1--11.
[14]
Pierre Bourque and Richard E Fairley. 2014. SWEBOK: guide to the software engineering body of knowledge. IEEE Computer Society.
[15]
Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beckwith, Irwin Kwan, Anicia Peters, and William Jernigan. 2016. GenderMag: A method for evaluating software's gender inclusiveness. Interacting with Computers, Vol. 28, 6 (2016), 760--787.
[16]
H Alperen cC etin, Emre Doug an, and Eray Tüzün. 2021. A review of code reviewer recommendation studies: Challenges and future directions. Science of Computer Programming (2021), 102652.
[17]
Curtis K Chan and Michel Anteby. 2016. Task segregation as a mechanism for within-job inequality: Women and men of the transportation security administration. Administrative Science Quarterly, Vol. 61, 2 (2016), 184--216.
[18]
Moataz Chouchen, Ali Ouni, Mohamed Wiem Mkaouer, Raula Gaikovina Kula, and Katsuro Inoue. 2021. WhoReview: A multi-objective search-based approach for code reviewers recommendation in modern code review. Applied Soft Computing, Vol. 100 (2021), 106908.
[19]
McKinsey & Company. 2021. Women in the Workplace. Available from texttthttps://www.mckinsey.com/featured-insights/diversity-and-inclusion/women-in-the-workplace.
[20]
Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. 2013. Development and deployment at facebook. IEEE Internet Computing, Vol. 17, 4 (2013), 8--17.
[21]
Leonardo B Furtado, Bruno Cartaxo, Christoph Treude, and Gustavo Pinto. 2021. How Successful Are Open Source Contributions From Countries With Different Levels of Human Development? IEEE Software, Vol. 38, 02 (2021), 58--63.
[22]
Manolis Galenianos. 2021. Referral networks and inequality. The Economic Journal, Vol. 131, 633 (2021), 271--301.
[23]
V Gewin. 2020. The time tax put on scientists of colour. Nature, Vol. 583, 7816 (2020), 479--481.
[24]
Google. 2021. Annual Diversity Report. https://diversity.google/annual-report/
[25]
Cassandra M Guarino and Victor MH Borden. 2017. Faculty service loads and gender: Are women taking care of the academic family? Research in higher education, Vol. 58, 6 (2017), 672--694.
[26]
Martine R Haas and Morten T Hansen. 2007. Different knowledge, different benefits: Toward a productivity perspective on knowledge sharing in organizations. Strategic management journal, Vol. 28, 11 (2007), 1133--1153.
[27]
Palak Halvadia and John Anvik. [n.,d.]. Code Reviewer Recommendation Systems: Past and Present. ( [n.,d.]).
[28]
Jun He and Lee A Freeman. 2010. Are men more technology-oriented than women? The role of gender on the development of general computer self-efficacy of college students. Journal of Information Systems Education, Vol. 21, 2 (2010), 203--212.
[29]
Madeline E Heilman. 1983. Sex bias in work settings: The lack of fit model. Research in organizational behavior (1983).
[30]
Stefan J Heitkamp, Stefan Rüttermann, and Susanne Gerhardt-Szép. 2018. Work shadowing in dental teaching practices: evaluation results of a collaborative study between university and general dental practices. BMC medical education, Vol. 18, 1 (2018), 1--14.
[31]
Laura E Hirshfield and Tiffany D Joseph. 2012. 'We need a woman, we need a black woman': Gender, race, and identity taxation in the academy. Gender and Education, Vol. 24, 2 (2012), 213--227.
[32]
Yu Huang, Kevin Leach, Zohreh Sharafi, Nicholas McKay, Tyler Santander, and Westley Weimer. 2020. Biases and differences in code review using medical imaging and eye-tracking: genders, humans, and machines. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 456--468.
[33]
Ann Hergatt Huffman, Jason Whetten, and William H Huffman. 2013. Using technology in higher education: The influence of gender roles on technology self-efficacy. Computers in Human Behavior, Vol. 29, 4 (2013), 1779--1786.
[34]
Nasif Imtiaz, Justin Middleton, Joymallya Chakraborty, Neill Robson, Gina Bai, and Emerson Murphy-Hill. 2019. Investigating the effects of gender bias on GitHub. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 700--711.
[35]
Ciera Jaspan, Matt Jorde, Carolyn D Egelman, Collin Green, Ben Holtz, Edward K Smith, Margaret Morrow Hodges, Andrea Knight, Elizabeth Kammer, Jillian Dicker, et al. 2020. Enabling the Study of Software Development Behavior with Cross-Tool Logs. IEEE Software, Vol. 37, 6 (2020), 44--51.
[36]
Ciera Jaspan, Matthew Jorde, Andrea Knight, Caitlin Sadowski, Edward K Smith, Collin Winter, and Emerson Murphy-Hill. 2018. Advantages and disadvantages of a monolithic repository: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, 225--234.
[37]
Mika Kivim"aki, Markus Jokela, Solja T Nyberg, Archana Singh-Manoux, Eleonor I Fransson, Lars Alfredsson, Jakob B Bjorner, Marianne Borritz, Hermann Burr, Annalisa Casini, et al. 2015. Long working hours and risk of coronary heart disease and stroke: a systematic review and meta-analysis of published and unpublished data for 603 838 individuals. The lancet, Vol. 386, 10005 (2015), 1739--1746.
[38]
Vladimir Kovalenko, Nava Tintarev, Evgeny Pasynkov, Christian Bird, and Alberto Bacchelli. 2018. Does reviewer recommendation help developers? IEEE Transactions on Software Engineering, Vol. 46, 7 (2018), 710--731.
[39]
Jane Margolis and Allan Fisher. 2002. Unlocking the clubhouse: Women in computing.
[40]
Susan Michie and Debra L Nelson. 2006. Barriers women face in information technology careers: Self-efficacy, passion and gender biases. Women in management review (2006).
[41]
Ehsan Mirsaeedi and Peter C Rigby. 2020. Mitigating turnover with code review recommendation: balancing expertise, workload, and knowledge distribution. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1183--1195.
[42]
Emerson Murphy-Hill, Jillian Dicker, Margaret Morrow Hodges, Carolyn D Egelman, Ciera Jaspan, Lan Cheng, Elizabeth Kammer, Ben Holtz, Matt Jorde, Andrea Knight, et al. 2021. Engineering Impacts of Anonymous Author Code Review: A Field Experiment. Transactions on Software Engineering (2021).
[43]
Reza Nadri, Gema Rodriguezperez, and Meiyappan Nagappan. 2021. On the Relationship Between the Developer's Perceptible Race and Ethnicity and the Evaluation of Contributions in OSS. IEEE Transactions on Software Engineering (2021).
[44]
Meredith Nash and Robyn Moore. 2019. 'I was completely oblivious to gender': an exploration of how women in STEMM navigate leadership in a neoliberal, post-feminist context. Journal of Gender Studies, Vol. 28, 4 (2019), 449--461.
[45]
Mogens J Pedersen and Vibeke L Nielsen. 2020. Bureaucratic decision-making: A multi-method study of gender similarity bias and gender stereotype beliefs. Public Administration, Vol. 98, 2 (2020), 424--440.
[46]
Soumaya Rebai, Abderrahmen Amich, Somayeh Molaei, Marouane Kessentini, and Rick Kazman. 2020. Multi-objective code reviewer recommendations: balancing expertise, availability and collaborations. Automated Software Engineering, Vol. 27, 3 (2020), 301--328.
[47]
José E Rodr'iguez, Kendall M Campbell, and Linda H Pololi. 2015. Addressing disparities in academic medicine: what of the minority tax? BMC Medical Education, Vol. 15, 1 (2015), 1--5.
[48]
Louise Marie Roth. 2011. Selling women short. Princeton University Press, Princeton, New Jersey.
[49]
Shade Ruangwan, Patanamon Thongtanunam, Akinori Ihara, and Kenichi Matsumoto. 2019. The impact of human factors on the participation decision of reviewers in modern code review. Empirical Software Engineering, Vol. 24, 2 (2019), 973--1016.
[50]
Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, Gothenburg, Sweden, 181--190.
[51]
Norbert K Semmer, Nicola Jacobshagen, Laurenz L Meier, Achim Elfering, Terry A Beehr, Wolfgang K"alin, and Franziska Tschan. 2015. Illegitimate tasks as a source of work stress. Work & Stress, Vol. 29, 1 (2015), 32--56.
[52]
Eva Skuratowicz and Larry W Hunter. 2004. Where do women's jobs come from? Job resegregation in an American bank. Work and occupations, Vol. 31, 1 (2004), 73--110.
[53]
Stack Overflow. 2021. 2021 Developer Survey: Demographics. Available from texttthttps://insights.stackoverflow.com/survey/2021#developer-profile-demographics.
[54]
Anton Strand, Markus Gunnarson, Ricardo Britto, and Muhmmad Usman. 2020. Using a context-aware approach to recommend code reviewers: findings from an industrial case study. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice. 1--10.
[55]
Josh Terrell, Andrew Kofink, Justin Middleton, Clarissa Rainear, Emerson Murphy-Hill, Chris Parnin, and Jon Stallings. 2017. Gender differences and bias in open source: Pull request acceptance of women versus men. PeerJ Computer Science, Vol. 3 (2017), e111.
[56]
Martin Tolich and Celia Briar. 1999. Just checking it out: exploring the significance of informal gender divisions amongst American supermarket employees. Gender, Work & Organization, Vol. 6, 3 (1999), 129--133.
[57]
Robert Kevin Toutkoushian and Marcia L Bellas. 1999. Faculty time allocations and research productivity: Gender, race and family effects. The review of higher education, Vol. 22, 4 (1999), 367--390.
[58]
Raymond Van Wijk, Justin JP Jansen, and Marjorie A Lyles. 2008. Inter-and intra-organizational knowledge transfer: a meta-analytic review and assessment of its antecedents and consequences. Journal of management studies, Vol. 45, 4 (2008), 830--853.
[59]
Joan C. Williams, Su Li, Roberta Rincon, and Peter Finn. 2016. Climate control: Gender and racial bias in engineering? Center for Worklife Law & Society of Women Engineers.
[60]
Joan C. Williams, Marina Multhaup, Su Li, and Rachel Korn. 2019. You Can't Change What You Can't See: Interrupting Racial and Gender Bias in the Legal Profession. American Bar Association and Minority Corporate Counsel Association.
[61]
Titus Winters, Tom Manshreck, and Hyrum Wright. 2020. Software engineering at google: Lessons learned from programming over time. O'Reilly Media, Newton, Massachusetts.
[62]
Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu. 2015. Wait for it: Determinants of pull request evaluation latency on github. In 2015 IEEE/ACM 12th working conference on mining software repositories. IEEE, Florence, Italy, 367--371.
[63]
Motahareh Bahrami Zanjani, Huzefa Kagdi, and Christian Bird. 2016. Automatically recommending peer reviewers in modern code review. IEEE Transactions on Software Engineering, Vol. 42, 6 (2016), 530--543.
Index Terms
- Systemic Gender Inequities in Who Reviews Code
Recommendations
Code reviews in open source projects : how do gender biases affect participation and outcomes?
AbstractContextContemporary software development organizations lack diversity, and the ratios of women in Free and Open-Source Software (FOSS) communities are even lower than the industry average. Although the results of recent studies hint at the ...
Understanding code snippets in code reviews: a preliminary study of the OpenStack community
ICPC '22: Proceedings of the 30th IEEE/ACM International Conference on Program ComprehensionCode review is a mature practice for software quality assurance in software development with which reviewers check the code that has been committed by developers, and verify the quality of code. During the code review discussions, reviewers and ...
Gender Equity in Computing: International Faculty Perceptions and Current Practices
ITiCSE '16: Proceedings of the 2016 ITiCSE Working Group ReportsIn many countries serious effort has been put into developing and running programs that encourage girls to enjoy learning programming. At school level, many girls have done very well in these experiences, but despite their confidence and enthusiasm for ...
Comments
Information & Contributors
Information
Published In
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 April 2023
Published in PACMHCI Volume 7, Issue CSCW1
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 528Total Downloads
- Downloads (Last 12 months)390
- Downloads (Last 6 weeks)36
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
Cited By
View all- Murali ASahu GThangarajah KZimmerman BRodríguez-Pérez GNagappan M(2024)Diversity in issue assignment: humans vs botsEmpirical Software Engineering10.1007/s10664-023-10424-629:2Online publication date: 9-Jan-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in