Article

Free access

A replicated assessment and comparison of common software cost modeling techniques

Authors:

Lionel C. Briand,

Tristen Langley,

Isabella WieczorekAuthors Info & Claims

ICSE '00: Proceedings of the 22nd international conference on Software engineering

Pages 377 - 386

https://doi.org/10.1145/337180.337223

Published: 01 June 2000 Publication History

Abstract

Delivering a software product on time, within budget, and to an agreed level of quality is a critical concern for many software organizations. Underestimating software costs can have detrimental effects on the quality of the delivered software and thus on a company's business reputation and competitiveness. On the other hand, overestimation of software cost can result in missed opportunities to funds in other projects. In response to industry demand, a myriad of estimation techniques has been proposed during the last three decades. In order to assess the suitability of a technique from a diverse selection, its performance and relative merits must be compared.

The current study replicates a comprehensive comparison of common estimation techniques within different organizational contexts, using data from the European Space Agency. Our study is motivated by the challenge to assess the feasibility of using multi-organization data to build cost models and the benefits gained from company-specific data collection. Using the European Space Agency data set, we investigated a yet unexplored application domain, including military and space projects. The results showed that traditional techniques, namely, ordinary least-squares regression and analysis of variance outperformed Analogy-based estimation and regression trees. Consistent with the results of the replicated study no significant difference was found in accuracy between estimates derived from company-specific data and estimates derived from multi-organizational data.

References

[1]

Boehm, B. Software Engineering Economics. Englewood Cliffs, NJ Prentice Hall (1981).

Digital Library

[2]

Boehm B., Clark, B., Horowitz, E. Westland, C. Cost models for future software life cycle processes: COCOMO 2.0. Annals of Software Engineering, 1 (1995), 57-94.

[3]

Bohrnstedt G., Carter, T. Robustness in Regression Analysis. In: Costner, H. (ed). Chapter 5, Sociological Methodology. Jossey-Bass (1971).

[4]

Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J. Classification and Regression Trees. Wadsworth & Books/Cole Advanced Books & Software (1984).

[5]

Briand, L.C., Basili, V.R., Thomas, W.M. A pattern recognition approach for software engineering data analysis. IEEE Transactions on Software Engineering, 18, 11 (1992) 931-942.

Digital Library

[6]

Briand, L.C., El Emam K., Bomarius, F. A Hybrid Method for Software Cost Estimation, Benchmarking, and Risk Assessment. Proceedings of the 20 th International Conference on Software Engineering, ICSE-20 (April 1998) 390-399.

Digital Library

[7]

Briand, L.C., El Emam, K., Wieczorek I. Explaining the Cost of European Space and Military Projects. In: Proceedings of the 21 st International Conference on Software Engineering, ICSE 99, (Los Angeles, USA 1999) 303-312.

Digital Library

[8]

Briand, L.C. El Emam K., Maxwell, K., Surmann, D., Wieczorek, I. An Assessment and Comparison of Common Software Cost Estimation Models. In: Proceedings of the 21 st International Conference on Software Engineering, ICSE 99, (Los Angeles, USA 1998) 313-322.

Digital Library

[9]

Briand, L.C., Langley, T., Wieczorek, I. Using the European Space Agency Database: A replicated Assessment of Common Software Cost Estimation Techniques. Technical Report, ISERN TR-99-15, International Software Engineering Research Network (1999).

[10]

CBR-Works, 4.0 beta. Research group "Artificial Intelligence- Knowledge-Based Systems", University of Kaiserslautern <http://wwwagr.informatik.uni-kl. de/-lsa/CBRatUKL.html>

[11]

Conte, S.D., Dunsmore, H.E., Shen, V. Y. Software engineering metrics and models. The Benjamin/Cummings Publishing Company, Inc. (1986).

Digital Library

[12]

Chulani S., Boehm B., Steece B. Bayesian Analysis of Empirical Software Engineering Cost Models. IEEE Transactions on Software Engineering, 25,4 (1999)

Digital Library

[13]

Davidson, R., McKinnon, J. Several Tests for Model Specification in the Presence of Alternative Hypotheses. Econometrica, 49, 3 (1981), 781-93.

[14]

Finnie, G. R., Wittig, G. E. A comparison of software effort estimation techniques: using function points with neural networks, case based reasoning and regression models. J. Systems Software, 39 (1997) 281-289.

Digital Library

[15]

Gibbons, J.D. S. Nonparametric Statistics. Series: Quantitative Application in the Social Sciences 90, SAGE University Paper (1993).

[16]

Gray, A., MacDonell, D. A comparison of techniques for developing predictive models of software metrics. Information and Software Technology, 39, (1997) 425-437

[17]

Greves, D., Schreiber, B. The ESA Initiative for Software Productivity Benchmarking and Effort Estimation. ESA Bulletin, 87 (August 1996). <http://esapub.esrin.esa.it/bulletin/bulett87/greves87.ht m>

[18]

H~st, M., Wohlin, C. An Experimental Study of Individual Subjective Effort Estimation and Combinations of the Estimates. In: Proceedings of the 20 st International Conference on Software Engineering, ICSE 98, (Japan, 1998) 332-339.

Digital Library

[19]

<http://dec.bournemouth.ac.uk/dec_ind/decind22/web/ Angel.html>.

[20]

Jones, C. Applied Software Measurement: Assuring Productivity and Quality, (2 nd Ed.), Mc-Graw-Hill, NY, USA, (1996).

Digital Library

[21]

J~rgensen, M. Experience with the accuracy of Software Maintenance Task Effort Prediction Models. IEEE Transactions on Software Engineering, 21, 8 (August, 1995) 674-681.

Digital Library

[22]

Kitchenham, B. A. Procedure for Analyzing Unbalanced Data Sets, IEEE Transactions on Software Engineering, 24, 4 (April 1998) 278-301.

Digital Library

[23]

Schroeder, L. Sjoquist, D., Stephan, P. Understanding Regression Analysis: An Introductory Guide. No. 57 In Series: Quantitative Applications in the Social Sciences, Sage Publications, Newbury Park CA, USA, (1986)

[24]

Maxwell, K., Van Wassenhove, L. and Dutta, S. Software Development Productivity of European Space, Military and Industrial Applications. IEEE Transactions on Software Engineering, 22, 10 (1996).

Digital Library

[25]

Shepperd, M., Schofield, C. Estimating software project effort using analogies. IEEE Transactions on Software Engineering, 23, 12 (November 1997) 736- 743.

Digital Library

[26]

Stensrud, E., Myrtveit, I. Human Performance Estimation with Analogy and Regression Models. In: Proceedings of the METRICS 98 Symposium, (1998) 205-213.

Digital Library

[27]

Spector, P. Ratings of Equal and unequal Response Choice Intervals. The Journal of Social Psychology, 112 (1980) 115-119.

[28]

Srinivasan, K., Fisher, D. Machine learning approaches to estimating software development effort. IEEE Transactions on Software Engineering, 21, 2 (February 1995) 126-137.

Digital Library

[29]

StataCorp, Stata Statistical Software: Release 5.0. Stata Corporation, College Station, (Texas 1997). <http://www.stata.com>

[30]

Steinberg, D., Colla, P. CART, Classification and Regression Trees, Tree Structured Non-parametric Data Analysis, Interface Documentation. Saflord Systems (1995), <http://www.salfordsystems.com/ index.html>

[31]

Walkerden F., Jeffery R. An Empirical Study of Analogy-based Software Effort Estimation. Empirical Software Engineering, 4, 2, (June 1999) 135-158.

Digital Library

[32]

Weiss, S., Kulikowski, C. Computer Systems that Learn. Morgan Kaufmann Publishers, Inc. San Francisco, CA, (1991).

Cited By

Feizpour ETahayori HSami A(2023)CoBRA without experts: New paradigm for software development effort estimation using COCOMO metricsJournal of Software: Evolution and Process10.1002/smr.2569Online publication date: 25-Apr-2023
https://doi.org/10.1002/smr.2569
Jaafari APazhouhan IBettinger P(2021)Machine Learning Modeling of Forest Road Construction CostsForests10.3390/f1209116912:9(1169)Online publication date: 28-Aug-2021
https://doi.org/10.3390/f12091169
Wani ZGiri KBashir R(2019)A Generic Data Mining Model for Software Cost Estimation Based on Novel Input Selection ProcedureInternational Journal of Information Retrieval Research10.4018/IJIRR.20190101029:1(16-32)Online publication date: 1-Jan-2019
https://doi.org/10.4018/IJIRR.2019010102
Show More Cited By

Index Terms

A replicated assessment and comparison of common software cost modeling techniques

Recommendations

An assessment and comparison of common software cost estimation modeling techniques
ICSE '99: Proceedings of the 21st international conference on Software engineering
Exploring the human and organizational aspects of software cost estimation
CHASE '10: Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering

Cost estimation is important for planning, scheduling, budgeting and pricing of software development. Previous research has mainly focused on improving estimates and the associated processes. However, there is still a lack of research on human and ...
Software Cost Estimation Inhibitors - A Case Study
PROFES '08: Proceedings of the 9th international conference on Product-Focused Software Process Improvement

Software cost estimation errors are increasing in the automotive industry along with the number of software components in modern vehicles. As the software cost estimation is an important and problematic part of project planning there is a need of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '00: Proceedings of the 22nd international conference on Software engineering

June 2000

843 pages

ISBN:1581132069

DOI:10.1145/337180

Chairmen:
Carlo Ghezzi
Politecnico di Milano, Milau, Italy
,
Mehdi Jazayeri
Technical Univ. of Vienna, Vienna, Austria
,
Alexander L. Wolf
Univ. of Colorado, Boulder, USA

Copyright © 2000 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Irish Comp Soc: Irish Computer Society
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2000

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE00

Sponsor:

Irish Comp Soc
SIGSOFT
IEEE-CS

ICSE00: 22nd International Conference on Software Engineering

June 4 - 11, 2000

Limerick, Ireland

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

114
Total Citations
View Citations
1,906
Total Downloads

Downloads (Last 12 months)78
Downloads (Last 6 weeks)8

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Feizpour ETahayori HSami A(2023)CoBRA without experts: New paradigm for software development effort estimation using COCOMO metricsJournal of Software: Evolution and Process10.1002/smr.2569Online publication date: 25-Apr-2023
https://doi.org/10.1002/smr.2569
Jaafari APazhouhan IBettinger P(2021)Machine Learning Modeling of Forest Road Construction CostsForests10.3390/f1209116912:9(1169)Online publication date: 28-Aug-2021
https://doi.org/10.3390/f12091169
Wani ZGiri KBashir R(2019)A Generic Data Mining Model for Software Cost Estimation Based on Novel Input Selection ProcedureInternational Journal of Information Retrieval Research10.4018/IJIRR.20190101029:1(16-32)Online publication date: 1-Jan-2019
https://doi.org/10.4018/IJIRR.2019010102
Korenaga KMonden AYucel Z(2019)Data Smoothing for Software Effort Estimation2019 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD)10.1109/SNPD.2019.8935841(501-506)Online publication date: Jul-2019
https://doi.org/10.1109/SNPD.2019.8935841
Mendes EFelizardo KWohlin CKalinowski M(2019)Search Strategy to Update Systematic Literature Reviews in Software Engineering2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA.2019.00061(355-362)Online publication date: Aug-2019
https://doi.org/10.1109/SEAA.2019.00061
Najm AZakrani AMarzak A(2019)Decision Trees Based Software Development Effort Estimation: A Systematic Mapping Study2019 International Conference of Computer Science and Renewable Energies (ICCSRE)10.1109/ICCSRE.2019.8807544(1-6)Online publication date: Jul-2019
https://doi.org/10.1109/ICCSRE.2019.8807544
Nguyen VHuynh TBoehm BHuang LTruong T(2019)Investigating the use of duration‐based windows and estimation by analogy for COCOMOJournal of Software: Evolution and Process10.1002/smr.217631:10Online publication date: 25-Oct-2019
https://dl.acm.org/doi/10.1002/smr.2176
Angelis LMittas NChatzipetrou P(2018)A Framework of Statistical and Visualization Techniques for Missing Data Analysis in Software Cost EstimationIntelligent Systems10.4018/978-1-5225-5643-5.ch014(345-372)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-5643-5.ch014
Angelis LMittas NChatzipetrou P(2018)A Framework of Statistical and Visualization Techniques for Missing Data Analysis in Software Cost EstimationComputer Systems and Software Engineering10.4018/978-1-5225-3923-0.ch017(433-460)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3923-0.ch017
Saif SWahid A(2018)Web Effort Estimation Using FP and WO: A Critical Study2018 Second International Conference on Computing Methodologies and Communication (ICCMC)10.1109/ICCMC.2018.8487472(357-361)Online publication date: Feb-2018
https://doi.org/10.1109/ICCMC.2018.8487472
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents