research-article

The role of outcome feedback in improving the uncertainty assessment of software development effort estimates

Authors:

Tanja M. Gruschke,

Magne JørgensenAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 17, Issue 4

Article No.: 20, Pages 1 - 35

https://doi.org/10.1145/13487689.13487693

Published: 25 August 2008 Publication History

Abstract

Previous studies report that software developers are over-confident in the accuracy of their effort estimates. Aim: This study investigates the role of outcome feedback, that is, feedback about the discrepancy between the estimated and the actual effort, in improving the uncertainty assessments. Method: We conducted two in-depth empirical studies on uncertainty assessment learning. Study 1 included five student developers and Study 2, 10 software professionals. In each study the developers repeatedly assessed the uncertainty of their effort estimates of a programming task, solved the task, and received estimation accuracy outcome feedback. Results: We found that most, but not all, developers were initially over-confident in the accuracy of their effort estimates and remained over-confident in spite of repeated and timely outcome feedback. One important, but not sufficient, condition for improvement based on outcome feedback seems to be the use of explicitly formulated, instead of purely intuition-based, uncertainty assessment strategies.

References

[1]

Bengtsson, C., Persson, M., and Willenhag, P. 2005. Gender and overconfidence. Economics Lett. 86, 2 (Feb.), 199--203.

[2]

Bolger, F. and Wright, G. 1994. Assessing the quality of expert judgment: Issues and analysis. Decision Support Syst. 11, 1, 1--24.

Digital Library

[3]

Bryan, G. E. 1994. Not All Programmers Are Created Equal. In Proceedings of the IEEE Aerospace Applications Conference, 55--62.

[4]

Buehler, R. and Griffin, D. 2003. Planning, personality, and prediction: The role of future focus in optimistic time predictions. Organiza. Behav. Hum. Decision Proc. 92, 80--90.

[5]

Buehler, R., Griffin, D., and Ross, M. 1994. Exploring the “Planning fallacy”: Why people underestimate their task completion times. J. Personality Soc. Psych. 67, 3, 366--381.

[6]

Buehler, R., Griffin, D., and MacDonald, H. 1997. The role of motivated reasoning in optimistic time predictions. Personality Soc. Psych. Bull. 23, 3, 238--247.

[7]

Connolly, T. and Dean, D. 1997. Decomposed versus holistic estimates of effort required for software writing tasks. Manag. Sci. 43, 7, 1029--1045.

Digital Library

[8]

Einhorn, H. J. and Hogarth, R. M. 1978. Confidence in judgment: Persistence of the illusion of validity. Psych. Rev. 85, 5, 395--416.

[9]

Ericson, K. A. and Krampe, R. T. 1993. The role of deliberate practice in the acquisition of expert performance. Psych. Rev. 100, 3, 363--406.

[10]

Gagne, R. M. and Medsker, K. L. 1996. The Conditions of Learning: Training Applications. Harcourt Brace.

[11]

Gigerenzer, G. and Hoffrage, U. 1995. How to improve Bayesian reasoning without instruction: Frequency formats. Psych. Rev. 102, 4, 684--704.

[12]

Gray, A., MacDonell, S. G., and Shepperd, M. J. 1999. Factors systematically associated with errors in subjective estimates of software development effort: the stability of expert judgment. In Proceedings of the IEEE 6th International Metrics Symposium, IEEE Computer Society, 216--227.

Digital Library

[13]

Green, K. W. J., Medlin, B., and Whitten, D. 2004. Developing optimism to improve performance: an approach for the manufacturing sector. Industrial Manag. Data Sys. 104, 2, 106--114.

[14]

Hammond, K. R. 1996. Human Judgement and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. Oxford University Press, Oxford, UK.

[15]

Harrison, W. 1997. Editorial. Empirical Softw. Engin. 2, 1, 7--10.

Digital Library

[16]

Heine, S. J. and Lehman, D. R. 1995. Cultural variation in unrealistic optimism: Does the West feel more invulnerable than the East&quest; J. Personality Soc. Psych. 68, 4, 595--607.

[17]

Hogarth, R. M. 2001. Educating Intuition. University of Chicago Press, Chicago.

[18]

Hughes, R. T. 1996. Expert judgement as an estimating method. Inform. Softw. Tech. 38, 2, 67--75.

[19]

Humphrey, W. S. 1996. The PSP and personal project estimating. Amer. Program. 9, 6, 2--15.

[20]

Johnson, D. D. P. 2004. Overconfidence and War. Harvard University Press, Cambridge, MA.

[21]

Jørgensen, M. 2004a. Top-down and bottom-up expert estimation of software development effort. Inform. Softw. Tech. 46, 1, 3--16.

[22]

Jørgensen, M. 2004b. A review of studies on expert estimation of software development effort. J. Syst. Softw. 70, 1--2, 37--60.

Digital Library

[23]

Jørgensen, M. 2004c. Realism in assessment of effort estimation uncertainty: It matters how you ask. IEEE Trans. Softw. Engin. 30, 4, 209--217.

Digital Library

[24]

Jørgensen, M. 2005. Evidence-Based Guidelines for assessment of software development cost uncertainty. IEEE Trans. Softw. Engin. 31, 11, 924--954.

Digital Library

[25]

Jørgensen, M. and Moløkken-Østvold, K. 2004a. Eliminating over-confidence in software development effort estimates. In Conference on Product Focused Software Process Improvement (PROFES) Springer-Verlag 174--184.

[26]

Jørgensen, M. and Moløkken-Østvold, K. 2004b. Reasons for software effort estimation error: impact of respondent role, information collection approach, and data analysis method. IEEE Trans. Softw. Engin. 30, 12, 993--1007.

Digital Library

[27]

Jørgensen, M. and Sjøberg, D. I. K. 2002. Impact of experience on maintenance skills. J. Softw. Main. Evol. Res. Practice 14, 2, 123--146.

Digital Library

[28]

Jørgensen, M. and Sjøberg, D. I. K. 2003a. An effort prediction interval approach based on the empirical distribution of previous estimation accuracy. Inform. Softw. Tech. 45, 3, 123--136.

[29]

Jørgensen, M. and Sjøberg. D. 2003b. Generalization and theory-building in software engineering research. In Proceedings of Empirical Assessment in Software Engineering (EASE'04) 29--36.

[30]

Jørgensen, M., Faugli, B., and Gruschke, T. M. Characteristics of Software Engineers with Optimistic Predictions. J. Syst. Softw. To appear.

[31]

Jørgensen, M., Indahl, U., and Sjøberg, D. 2003. Software effort estimation by analogy and regression toward the mean. J. Syst. Softw. 68, 3, 253--262.

Digital Library

[32]

Jørgensen, M., Teigen, K. H., and Moløkken, K. 2004. Better sure than safe&quest; Over-confidence in judgement based software development effort prediction intervals. J. Syst. Softw. 70, 1-2, 79--93.

Digital Library

[33]

Kitchenham, B., Pfleeger, S. L., McColl, B., and Eagan, S. 2002. An empirical study of maintenance and development estimation accuracy. J. Syst. Softw. 64, 1, 57--77.

Digital Library

[34]

Klayman, J., Soll, J. B., González-Vallejo, C., and Barlas, S. 1999. Overconfidence: It depends on how, what and whom you ask. Organizat. Behav. Hum. Decision Proc. 79, 3, 216--247.

[35]

Kruglanski, A. W. 1996. Motivated social cognition: Principles of the interface. In Social Psychology: Handbook of Basic Principles, E.T. Higgins and A. W. Kurglanski, Eds. Guilford Press, New York, 493--520.

[36]

Li, H., Shen, Q. P., and Love, P. E. D. 2005. Cost modelling of office buildings in Hong Kong: An exploratory study. Facilities 23, 9/10, 438--452.

[37]

Li, S. and Fang, Y. 2004. Respondents in Asian cultures (e.g., Chinese) are more risk-seeking and more overconfident than respondents in other cultures (e.g., in United States) but the reciprocal predictions are in total opposition: how and why&quest; J. Cognition Culture 4, 2, 263--292.

[38]

Lichtenstein, S. and Fischhoff, B. 1977. Do those who know more also know more about how much they know&quest; Organizat. Behav. Hum. Decision Proc. 20, 2, 159--183.

[39]

Manove, M. 1995. Entrepeneurs, optimism and the competitive edge. UFAE and IAE Working Papers 296.95,

[40]

Moløkken, K. and Jørgensen, M. 2003. A review of software surveys on software effort estimation. In Proceedings of the International Symposium on Empirical Software Engineering. IEEE Computer Society, 223--230.

Digital Library

[41]

Moløkken, K. and Jørgensen, M. 2005. Expert estimation of Web-development projects: Are software professionals in technical roles more optimistic than those in non-technical roles&quest; Empirical Softw. Engin. 10, 1, 7--30.

Digital Library

[42]

Moløkken-Østvold, K. and Jørgensen, M. 2004. Group processes in software effort estimation. Empirical Softw. Engin. 9, 4, 315--334.

Digital Library

[43]

Oskamp, S. 1965. Overconfidence in case-study judgments. J. Consult. Psych. 29, 3, 261--265.

[44]

Pietrasanta, A. M. 1968. Current methodological research. In Proceedings of the 23rd ACM National Conference, 341--346.

Digital Library

[45]

Prechelt, L. and Unger, B. 2001. An experiment measuring the effects of personal software process (PSP) training. IEEE Trans. Softw. Engin. 27, 5, 465--472.

Digital Library

[46]

Puncochar, J. M. and Fox, P. W. 2004. Confidence in individual and group decision making: When “two heads” are worse than one. J. Educat. Psych. 96, 3, 582--591.

[47]

Rozenblit, L. and Keil, F.C. 2002. The misunderstood limits of folk science: an illusion of explanatory depth. Cognitive Sci. 26, 5, 521--562.

[48]

Shadish, W. R., Cook, T. D., and Campbell, D.T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin.

[49]

Shenhar, A. J. 2001. One size does not fit all projects: Exploring classical contingency domains. Manag. Sci. 47, 3, 394--414.

Digital Library

[50]

Shepperd, J. A., Carroll, P., Grace, J., and Terry, M. 2002. Exploring the causes of comparative optimism. Psychologica Belgica 42, 1/2, 65--98.

[51]

Shepperd, J. A., Fernandez, J. K. and Quellette, J. A. 1996. Abandoning unrealistic optimism: Performance estimates and the temporal proximity of self-relevant feedback. J. Personality Social Psych. 70, 4, 844--855.

[52]

Steen, v. d. E. 2004. Rational overoptimism (and other biases). Amer. Econ. Rev. 94, 4, 1141--1151.

[53]

Webster Jr, M. 1994. Experimental methods. In Group Processes, M. Foschi and E. J. Lawler, Eds., Nelson-Hall Publishers, 43--69.

[54]

Whitecotton, S. M., Sanders, D. E., and Norris, K. B. 1998. Improving predictive accuracy with a combination of human intuition and mechanical decision aids. Organizat. Behav. Hum. Decision Proces. 76, 3, 325--348.

[55]

Wilson, L. F. and Brekke, N. 1994. Mental contamination and mental correction: unwanted influence on judgments and evaluation. Psych. Bull. 116, 1, 117--142.

[56]

Zendler, A. E., Horn, E., Schwartzel, H., and Plodererder, E. 2001. Demonstrating the usage of single-case designs in experimental software engineering. Inform. Softw. Tech. 43, 12, 681--691.

Cited By

Ahmetoglu YBrumby DCox A(2024)Bridging the Gap Between Time Management Research and Task Management App Design: A Study on the Integration of Planning Fallacy Mitigation StrategiesProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663404(1-14)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663404
Goswami IUrminsky O(2023)More time, more work: How time limits bias estimates of task scope and project durationJudgment and Decision Making10.1017/S193029750000819615:6(994-1008)Online publication date: 1-Jan-2023
https://doi.org/10.1017/S1930297500008196
Koval MJansen Y(2022)Do You See What You Mean? Using Predictive Visualizations to Reduce Optimism in Duration EstimatesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502010(1-19)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502010
Show More Cited By

Index Terms

The role of outcome feedback in improving the uncertainty assessment of software development effort estimates
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
  2. Software notations and tools
    1. Software configuration management and version control systems

Recommendations

Evaluating probabilistic software development effort estimates: Maximizing informativeness subject to calibration
Abstract Context
Probabilistic effort estimates inform about the uncertainty and may give useful input to plans, budgets and investment analyses.
Objective & method
This paper introduces, motivates and ...
Relative estimates of software development effort: Are they more accurate or less time-consuming to produce than absolute estimates, and to what extent are they person-independent?
Abstract Context
Estimates of software development effort may be given as judgments of relationships between the use of efforts on different tasks-that is, as relative estimates. The use of relative estimates has increased with the ...
Trust yourself! Or maybe not: factors related to overconfidence and uncertainty assessments of software effort estimates
SBES '21: Proceedings of the XXXV Brazilian Symposium on Software Engineering

Software effort estimates are uncertain, given that they are probabilistic assessments of the future. Evaluating their uncertainty involves assigning them an appropriate confidence level and is paramount for satisfying commitments in software projects. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 17, Issue 4

August 2008

139 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/13487689

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2008

Accepted: 01 August 2007

Revised: 01 November 2006

Received: 01 February 2006

Published in TOSEM Volume 17, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
1,012
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)2

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ahmetoglu YBrumby DCox A(2024)Bridging the Gap Between Time Management Research and Task Management App Design: A Study on the Integration of Planning Fallacy Mitigation StrategiesProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663404(1-14)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663404
Goswami IUrminsky O(2023)More time, more work: How time limits bias estimates of task scope and project durationJudgment and Decision Making10.1017/S193029750000819615:6(994-1008)Online publication date: 1-Jan-2023
https://doi.org/10.1017/S1930297500008196
Koval MJansen Y(2022)Do You See What You Mean? Using Predictive Visualizations to Reduce Optimism in Duration EstimatesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502010(1-19)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502010
Sarro FMoussa RPetrozziello AHarman M(2022)Learning From Mistakes: Machine Learning Enhanced Human Expert Effort EstimatesIEEE Transactions on Software Engineering10.1109/TSE.2020.304079348:6(1868-1882)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1109/TSE.2020.3040793
Matsubara PGadelha BSteinmacher IConte T(2022)SEXTAMTJournal of Systems and Software10.1016/j.jss.2021.111148185:COnline publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.jss.2021.111148
Matsubara PSteinmacher IMaldonado JGadelha BConte T(2021)Trust yourself! Or maybe not: factors related to overconfidence and uncertainty assessments of software effort estimatesProceedings of the XXXV Brazilian Symposium on Software Engineering10.1145/3474624.3474643(452-461)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3474624.3474643
Valdés-Souto FNaranjo-Albarrán L(2021)Improving the Software Estimation Models Based on Functional Size through Validation of the Assumptions behind the Linear Regression and the Use of the Confidence Intervals When the Reference Database Presents a Wedge-Shape FormProgramming and Computer Software10.1134/S036176882108025947:8(673-693)Online publication date: 28-Dec-2021
https://doi.org/10.1134/S0361768821080259
Goswami IUrminsky O(2021)Don’t fear the meter: How longer time limits bias managers to prefer hiring with flat fee compensationOrganizational Behavior and Human Decision Processes10.1016/j.obhdp.2020.10.016162(42-58)Online publication date: Jan-2021
https://doi.org/10.1016/j.obhdp.2020.10.016
Jorgensen M(2018)Looking Back on Previous Estimation Error as a Method to Improve the Uncertainty Assessment of Benefits and Costs of Software Development Projects2018 9th International Workshop on Empirical Software Engineering in Practice (IWESEP)10.1109/IWESEP.2018.00012(19-24)Online publication date: Dec-2018
https://doi.org/10.1109/IWESEP.2018.00012
Halkjelsvik TJørgensen MHalkjelsvik TJørgensen M(2018)Uncertainty of Time PredictionsTime Predictions10.1007/978-3-319-74953-2_6(71-79)Online publication date: 1-Mar-2018
https://doi.org/10.1007/978-3-319-74953-2_6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents