research-article

Trust yourself! Or maybe not: factors related to overconfidence and uncertainty assessments of software effort estimates

Authors:

Patricia Matsubara,

Igor Steinmacher,

José Maldonado,

Tayana ConteAuthors Info & Claims

SBES '21: Proceedings of the XXXV Brazilian Symposium on Software Engineering

Pages 452 - 461

https://doi.org/10.1145/3474624.3474643

Published: 05 October 2021 Publication History

Abstract

Software effort estimates are uncertain, given that they are probabilistic assessments of the future. Evaluating their uncertainty involves assigning them an appropriate confidence level and is paramount for satisfying commitments in software projects. However, estimators tend to be overconfident about their estimates, hampering the accuracy of their uncertainty assessments. Our research goal is to identify the factors related to overconfidence and uncertainty assessments in software estimation. To do so, we carried out a Systematic Literature Mapping (SLM), based on automated and snowballing searches. Our findings include eight factors related to overconfidence and uncertainty assessment. Some of them resulted in unexpected implications for practice. We also identified valuable and easy-to-use metrics that software practitioners can apply smoothly in their daily practice. Additionally, very few field and respondent studies exist about the topic. The software engineering area can significantly benefit from investigating how much practitioners know about the overconfidence effect, as well as of a better comprehension of the perceived importance, practices, and accuracy of uncertainty assessments in the software industry.

Supplementary Material

p452-matsubara-supplements (p452-matsubara-supplements.zip)

Supplemental files

Download
724.73 KB

References

[1]

Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou, Marijn Verbeek, and Alexander Chatzigeorgiou. 2019. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology 106 (Feb. 2019), 201–230. https://doi.org/10.1016/j.infsof.2018.10.006

[2]

Jorge Aranda and Steve Easterbrook. 2005. Anchoring and Adjustment in Software Estimation. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering(ESEC/FSE-13). ACM, New York, NY, USA, 346–355. https://doi.org/10.1145/1081706.1081761 event-place: Lisbon, Portugal.

Digital Library

[3]

Tharwon Arnuphaptrairong. 2018. The State of Practice of Software Cost Estimation: Evidence From Thai Software Firms. In Proceedings of the International MultiConference of Engineers and Computer Scientists, Vol. 2. Hong Kong, 6.

[4]

Dirk Basten and Ali Sunyaev. 2014. A Systematic Mapping of Factors Affecting Accuracy of Software Development Effort Estimation. Communications of the Association for Information Systems 34 (2014). https://doi.org/10.17705/1CAIS.03404

[5]

Ricardo Britto, Emilia Mendes, and Jürgen Börstler. 2015. An Empirical Investigation on Effort Estimation in Agile Global Software Development. In 2015 IEEE 10th International Conference on Global Software Engineering. 38–45. https://doi.org/10.1109/ICGSE.2015.10 ISSN: 2329-6313.

[6]

Carlos Eduardo Carbonera, Kleinner Farias, and Vinicius Bischoff. 2020. Software development effort estimation: a systematic mapping study. IET Software 14, 4 (2020), 328–344. https://doi.org/10.1049/iet-sen.2018.5334 _eprint: https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/iet-sen.2018.5334.

[7]

Stein Grimstad, Magne Jørgensen, and Kjetil Moløkken-Østvold. 2006. Software effort estimation terminology: The tower of Babel. Information and Software Technology 48, 4 (April 2006), 302–310. https://doi.org/10.1016/j.infsof.2005.04.004

Digital Library

[8]

Tanja M. Gruschke and Magne Jørgensen. 2008. The role of outcome feedback in improving the uncertainty assessment of software development effort estimates. ACM Transactions on Software Engineering and Methodology 17, 4 (Aug. 2008), 20:1–20:35. https://doi.org/10.1145/13487689.13487693

Digital Library

[9]

Torleif Halkjelsvik and Magne Jørgensen. 2018. Time Predictions: Understanding and Avoiding Unrealism in Project Planning and Everyday Life. Springer International Publishing. https://doi.org/10.1007/978-3-319-74953-2

[10]

Jo Erskine Hannay, Hans Christian Benestad, and Kjetil Strand. 2019. Agile Uncertainty Assessment for Benefit Points and Story Points. IEEE Software 36, 4 (July 2019), 50–62. https://doi.org/10.1109/MS.2018.2875845 Conference Name: IEEE Software.

[11]

M. Jorgensen. 2004. Realism in assessment of effort estimation uncertainty: it matters how you ask. IEEE Transactions on Software Engineering 30, 4 (April 2004), 209–217. https://doi.org/10.1109/TSE.2004.1274041 Conference Name: IEEE Transactions on Software Engineering.

Digital Library

[12]

M. Jorgensen. 2005. Evidence-based guidelines for assessment of software development cost uncertainty. IEEE Transactions on Software Engineering 31, 11 (Nov. 2005), 942–954. https://doi.org/10.1109/TSE.2005.128 Conference Name: IEEE Transactions on Software Engineering.

Digital Library

[13]

M. Jorgensen. 2005. Practical guidelines for expert-judgment-based software effort estimation. IEEE Software 22, 3 (May 2005), 57–63. https://doi.org/10.1109/MS.2005.73

Digital Library

[14]

Magne Jorgensen. 2014. What We Do and Don’t Know about Software Development Effort Estimation. IEEE Software 31, 2 (2014), 37–40. https://doi.org/10.1109/MS.2014.49

[15]

Magne Jørgensen. 2010. Identification of more risks can lead to increased over-optimism of and over-confidence in software development effort estimates. Information and Software Technology 52, 5 (May 2010), 506–516. https://doi.org/10.1016/j.infsof.2009.12.002

Digital Library

[16]

Magne Jørgensen. 2014. Communication of Software Cost Estimates. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering(EASE ’14). ACM, New York, NY, USA, 28:1–28:5. https://doi.org/10.1145/2601248.2601262 event-place: London, England, United Kingdom.

Digital Library

[17]

Magne Jørgensen. 2014. The Ignorance of Confidence Levels in Minimum-Maximum Software Development Effort Intervals. Lecture Notes on Software Engineering 2, 4 (2014), 327–330. https://doi.org/10.7763/LNSE.2014.V2.144

[18]

Magne Jørgensen. 2016. The Use of Precision of Software Development Effort Estimates to Communicate Uncertainty. In Software Quality. The Future of Systems and Software Development(Lecture Notes in Business Information Processing), Dietmar Winkler, Stefan Biffl, and Johannes Bergsmann (Eds.). Springer International Publishing, Cham, 156–168. https://doi.org/10.1007/978-3-319-27033-3_11

[19]

M. Jørgensen. 2018. Looking Back on Previous Estimation Error as a Method to Improve the Uncertainty Assessment of Benefits and Costs of Software Development Projects. In 2018 9th International Workshop on Empirical Software Engineering in Practice (IWESEP). 19–24. https://doi.org/10.1109/IWESEP.2018.00012 ISSN: 2573-2021.

[20]

Magne Jørgensen, Bjørn Faugli, and Tanja Gruschke. 2007. Characteristics of software engineers with optimistic predictions. Journal of Systems and Software 80, 9 (Sept. 2007), 1472–1482. https://doi.org/10.1016/j.jss.2006.09.047

Digital Library

[21]

Magne Jørgensen and Tanja M. Gruschke. 2009. The Impact of Lessons-Learned Sessions on Effort Estimation and Uncertainty Assessments. IEEE Transactions on Software Engineering 35, 3 (May 2009), 368–383. https://doi.org/10.1109/TSE.2009.2

Digital Library

[22]

Magne Jørgensen and Kjetil Moløkken. 2002. Combination of software development effort prediction intervals: why, when and how?. In Proceedings of the 14th international conference on Software engineering and knowledge engineering(SEKE ’02). ACM, Ischia, Italy, 425–428. https://doi.org/10.1145/568760.568833

Digital Library

[23]

Magne Jørgensen and Kjetil Moløkken. 2004. Eliminating Over-Confidence in Software Development Effort Estimates. In Product Focused Software Process Improvement(Lecture Notes in Computer Science), Frank Bomarius and Hajimu Iida (Eds.). Springer, Berlin, Heidelberg, 174–184. https://doi.org/10.1007/978-3-540-24659-6_13

[24]

Magne Jørgensen and Karl-Halvor Teigen. 2002. Uncertainty Intervals Versus Interval Uncertainty: an Alternative Method for Eliciting Effort Prediction Intervals in Software Development Projects. Singapore, 343–352.

[25]

Magne Jørgensen, Karl Halvor Teigen, and Kjetil Moløkken. 2004. Better sure than safe? Over-confidence in judgement based software development effort prediction intervals. Journal of Systems and Software 70, 1 (Feb. 2004), 79–93. https://doi.org/10.1016/S0164-1212(02)00160-7

Digital Library

[26]

Barbara Kitchenham and Stephen Linkman. 1997. Estimates, Uncertainty, and Risk. IEEE Software 14, 3 (May 1997), 69–74. https://doi.org/10.1109/52.589239

Digital Library

[27]

Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidence-Based Software Engineering and Systematic Reviews (1 ed.). CRC Press.

[28]

T. Little. 2006. Schedule estimation and uncertainty surrounding the cone of uncertainty. IEEE Software 23, 3 (May 2006), 48–54. https://doi.org/10.1109/MS.2006.82 Conference Name: IEEE Software.

Digital Library

[29]

Ana Magazinius, Sofia Börjesson, and Robert Feldt. 2012. Investigating intentional distortions in software cost estimation – An exploratory study. Journal of Systems and Software 85, 8 (Aug. 2012), 1770–1781. https://doi.org/10.1016/j.jss.2012.03.026

Digital Library

[30]

A. Magazinius and R. Feldt. 2011. Confirming Distortional Behaviors in Software Cost Estimation Practice. In 2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications. 411–418. https://doi.org/10.1109/SEAA.2011.61

[31]

Ana Magazinovic and Joakim Pernstål. 2008. Any other cost estimation inhibitors?. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM ’08. ACM Press, Kaiserslautern, Germany, 233. https://doi.org/10.1145/1414004.1414042

Digital Library

[32]

Patricia Matsubara, Igor Steinmacher, José Carlos Maldonado, Bruno Gadelha, and Tayana Conte. [n.d.]. Supplementary material for Trust Yourself!https://www.doi.org/10.6084/m9.figshare.14569230

[33]

Steve McConnell. 2006. Software Estimation: Demystifying the Black Art (1 ed.). Microsoft Press, Redomnd, Washington, USA.

[34]

K. Molokken and M. Jorgensen. 2003. A review of software surveys on software effort estimation. In 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings.223–230. https://doi.org/10.1109/ISESE.2003.1237981

[35]

Don A. Moore and Paul J. Healy. 2008. The trouble with overconfidence. Psychological Review 115, 2 (2008), 502–517. https://doi.org/10.1037/0033-295X.115.2.502 Place: US Publisher: American Psychological Association.

[36]

Don A. Moore and Derek Schatz. 2017. The three faces of overconfidence. Social and Personality Psychology Compass 11, 8 (2017), 1–12. https://doi.org/10.1111/spc3.12331 Place: United Kingdom Publisher: Wiley-Blackwell Publishing Ltd.

[37]

Ursula Passing and Martin Shepperd. 2003. An Experiment on Software Project Size and Effort Estimation. In Proceedings of the 2003 International Symposium on Empirical Software Engineering(ISESE ’03). IEEE Computer Society, Washington, DC, USA, 120–. http://dl.acm.org/citation.cfm?id=942801.943632

Digital Library

[38]

Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (Aug. 2015), 1–18.

Digital Library

[39]

Valerie F. Reyna, Priscila G. Brust-Renck, and Rebecca B. Weldon. 2021. Judgment and Decision Making. https://doi.org/10.1093/acrefore/9780190236557.013.536 ISBN: 9780190236557.

[40]

Sumeet Kaur Sehra, Yadwinder Singh Brar, Navdeep Kaur, and Sukhjit Singh Sehra. 2017. Research patterns and trends in software effort estimation. Information and Software Technology 91 (Nov. 2017), 1–21. https://doi.org/10.1016/j.infsof.2017.06.002

[41]

Martin Shepperd, Carolyn Mair, and Magne Jørgensen. 2018. An Experimental Evaluation of a De-biasing Intervention for Professional Software Developers. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing(SAC ’18). ACM, New York, NY, USA, 1510–1517. https://doi.org/10.1145/3167132.3167293 event-place: Pau, France.

Digital Library

[42]

Margaret-Anne Storey, Neil A. Ernst, Courtney Williams, and Eirini Kalliamvakou. 2020. The who, what, how of software engineering research: a socio-technical framework. Empirical Software Engineering 25, 5 (Sept. 2020), 4097–4129. https://doi.org/10.1007/s10664-020-09858-z

Digital Library

[43]

Adam Trendowicz, Jürgen Münch, and Ross Jeffery. 2011. State of the Practice in Software Effort Estimation: A Survey and Literature Review. In Software Engineering Techniques(Lecture Notes in Computer Science), Zbigniew Huzar, Radek Koci, Bertrand Meyer, Bartosz Walter, and Jaroslav Zendulka (Eds.). Springer Berlin Heidelberg, 232–245.

[44]

Susan Tyler. 2020. Chapter 1: How We Use Our Expectations. In Human Behavior and the Social Environment I. University of Arkansas Libraries. https://uark.pressbooks.pub/hbse1/chapter/chapter-1/

[45]

Muhammad Usman, Emilia Mendes, and Jürgen Börstler. 2015. Effort estimation in agile software development: a survey on the state of the practice. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering(EASE ’15). Association for Computing Machinery, Nanjing, China, 1–10. https://doi.org/10.1145/2745802.2745813

Digital Library

[46]

Muhammad Usman, Kai Petersen, Jürgen Börstler, and Pedro Santos Neto. 2018. Developing and using checklists to improve software effort estimation: A multi-case study. Journal of Systems and Software 146 (Dec. 2018), 286–309. https://doi.org/10.1016/j.jss.2018.09.054

[47]

Titus Winters, Tom Manshreck, and Hyrum Wrigth. 2020. What is Software Engineering?In Software Engineering at Google(1st ed.). O’Reilly Media, Inc.

[48]

Ilan Yaniv and Dean P. Foster. 1995. Graininess of judgment under uncertainty: An accuracy-informativeness trade-off. Journal of Experimental Psychology: General 124, 4 (1995), 424–432. https://doi.org/10.1037/0096-3445.124.4.424 Place: US Publisher: American Psychological Association.

[49]

He Zhang, Muhammad Ali Babar, and Paolo Tell. 2011. Identifying relevant studies in software engineering. Information and Software Technology 53, 6 (June 2011), 625–637. https://doi.org/10.1016/j.infsof.2010.12.010

Digital Library

Cited By

Kopp OSnethlage CSchwentker C(2023)JabRef: BibTeX-based literature management softwareTUGboat10.47397/tb/44-3/tb138kopp-jabref44:3(441-447)Online publication date: 2023
https://doi.org/10.47397/tb/44-3/tb138kopp-jabref
Bhardwaz SGodha R(2023)Svelte.js: The Most Loved Framework Today2023 2nd International Conference for Innovation in Technology (INOCON)10.1109/INOCON57975.2023.10101104(1-7)Online publication date: 3-Mar-2023
https://doi.org/10.1109/INOCON57975.2023.10101104
Sarpiri MAghaei MGandomani T(2023)Towards Better Software Development Effort Estimation with Analogy-based Approach and Nature-based Algorithms2023 International Conference on Information Technology (ICIT)10.1109/ICIT58056.2023.10225939(114-117)Online publication date: 9-Aug-2023
https://doi.org/10.1109/ICIT58056.2023.10225939

Recommendations

The role of outcome feedback in improving the uncertainty assessment of software development effort estimates

Previous studies report that software developers are over-confident in the accuracy of their effort estimates. Aim: This study investigates the role of outcome feedback, that is, feedback about the discrepancy between the estimated and the actual effort,...
Assessing Uncertainty of Software Development Effort Estimates: The Learning from Outcome Feedback
METRICS '05: Proceedings of the 11th IEEE International Software Metrics Symposium

To enable properly sized software projects budgets and plans it is important to be able to assess the uncertainty of the estimates of most likely effort required to complete the projects. Previous studies show that software professionals tend to be too ...
Uncertainty management in software effort estimation using a consistent fuzzy analogy-based method
Highlights
- New C-FASEE method is proposed to better managing for uncertainty in software effort estimation environment.
Abstract
Software effort estimation is a critical task in software project development management. Unfortunately, the uncertainty and inaccuracy are inherent properties of the software effort estimation environment. These are caused by the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBES '21: Proceedings of the XXXV Brazilian Symposium on Software Engineering

September 2021

473 pages

ISBN:9781450390613

DOI:10.1145/3474624

Copyright © 2021 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Samsung
Fundação de Amparo à Pesquisa do Estado do Amazonas
Coordenação de Aperfeiçoamento de Pessoal de Nível Superiorn
Conselho Nacional de Desenvolvimento Científico e Tecnológico

Conference

SBES '21

SBES '21: Brazilian Symposium on Software Engineering

September 27 - October 1, 2021

Joinville, Brazil

Acceptance Rates

Overall Acceptance Rate 147 of 427 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
109
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)5

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kopp OSnethlage CSchwentker C(2023)JabRef: BibTeX-based literature management softwareTUGboat10.47397/tb/44-3/tb138kopp-jabref44:3(441-447)Online publication date: 2023
https://doi.org/10.47397/tb/44-3/tb138kopp-jabref
Bhardwaz SGodha R(2023)Svelte.js: The Most Loved Framework Today2023 2nd International Conference for Innovation in Technology (INOCON)10.1109/INOCON57975.2023.10101104(1-7)Online publication date: 3-Mar-2023
https://doi.org/10.1109/INOCON57975.2023.10101104
Sarpiri MAghaei MGandomani T(2023)Towards Better Software Development Effort Estimation with Analogy-based Approach and Nature-based Algorithms2023 International Conference on Information Technology (ICIT)10.1109/ICIT58056.2023.10225939(114-117)Online publication date: 9-Aug-2023
https://doi.org/10.1109/ICIT58056.2023.10225939

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents