Article

Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles

Authors:

Wilfried GrossmannAuthors Info & Claims

ICSE '01: Proceedings of the 23rd International Conference on Software Engineering

Pages 145 - 154

Published: 01 July 2001 Publication History

Publisher Site Get Access

Abstract

Defect content estimation techniques (DCETs), based on defect data from inspection, estimate the total number of defects in a document to evaluate the development process. For inspections that yield few data points DCETs reportedly underestimate the number of defects. If there is a second inspection cycle, the additional defect data is expected to increase estimation accuracy.

In this paper we consider 3 scenarios to combine data sets from the inspection-reinspection process. We evaluate these approaches with data from an experiment in a university environment where 31 teams inspected and reinspected a software requirements document.

Main findings of the experiment were that reinspection data improved estimation accuracy. With the best combination approach all examined estimators yielded on average estimates within 20% around the true value, all estimates stayed within 40% around the true value.

References

[1]

V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Soerumgaard, and M. Zelkowitz, "The Empirical Investigation of Perspective-Based Reading." Empirical Software Engineering: An International Journal 1, 2 (1996), pp. 133-164.

[2]

S.Biffl,M.Halling,andM.K~hle, "Investigating the Effect of a Second Software Inspection Cycle" in Proc. APAQS 2000 Asia-Pacific Conference on Quality Software (Hong Kong, Oct. 2000), IEEE Comp.Soc.Press.

Digital Library

[3]

S. Biffl, "Using Inspection Data for Defect Estimation." IEEE Software special issue on recent project estimation methods, December 2000.

Digital Library

[4]

G.E.P. Box, W.G. Hunter, and J.S. Hunter, Statistics for Experimenters, John Wiley & Sons, 1978.

[5]

L.Briand,K.ElEmam,B.Freimut,and O,Laitenberger, "A Comprehensive Evaluation of Capture- Recapture Models for Estimating Software Defect Content", IEEE Transactions on Software Engineering, vol. 26, no. 6, June 2000, pp. 518-540.

Digital Library

[6]

A. Chao, "Estimating the Population Size for Capture-Recapture Data with Unequal Catchability." Biometrics 43 (Dec. 1987), pp. 783-791.

[7]

A. Chao, "Estimating Population Size for Sparse Data in Capture-Recapture Experiments." Biometrics 45 (June 1989), pp. 427-438.

[8]

A. Chao, S-M. Lee, and S-L. Jeng, "Estimating Population Size for Capture-Recapture Data when Capture Probabilities Vary by Time and Individual Animal." Biometrics 48 (March 1992), pp. 201-216.

[9]

B. Curtis, "By the Way, Did Anyone Study any Real Programmers?" Empirical Studies of Programmers: First Workshop (1986), Ablex Publishing Corporation, pp. 256-262.

Digital Library

[10]

St. Eick, C. Loader, M.D. Long, L.G. Votta, and S. Vander Wiel, "Estimating Software Fault Content before Coding." in Proc. 14 th Int. Conf. on Software Engineering (1992), pp. 59-65.

Digital Library

[11]

T. Gilb, and D. Graham. Software Inspection, Addison-Wesley, 1993.

Digital Library

[12]

O. Laitenberger, and J.M. DeBaud, "An Encompassing Life-Cycle Centric Survey of Software Inspection." Journal of Systems and Software 1, 51(2000).

Digital Library

[13]

J. Miller, M. Wood, and M. Roper, "Further experiences with scenarios and checklists." Empirical Software Engineering Journal 3, 1(1998), pp. 37-64.

Digital Library

[14]

H. Petersson, and C. Wohlin, "Evaluation of using Capture-Recapture Methods in Software Review Data." in Proc. Conference on Empirical Assessment and Evaluation in Software Engineering (Keele University, Staffordshire, UK, 1999).

[15]

H. Petersson, and C. Wohlin, "Evaluating Defect Content Estimation Rules in Software Inspections." in Proc. Conference on Empirical Assessment and Evaluation in Software Engineering (Keele University, Staffordshire, UK, 2000).

[16]

D.L. Otis, K.P. Burnham, G.C. White, and D.R. Anderson, 'Statistical Interference from Capture Data on closed animal populations." A Publication of the Wildlife Society, Wildlife Monographs 62 (1978).

[17]

C. Robson, Real World Research, Blackwell, 1993.

[18]

T. Thelin, and P. Runeson, "Fault Content Estimations using Extended Curve Fitting Models and Model Selection." in Proc. Conference on Empirical Assessment and Evaluation in Software Engineering (Keele University, Staffordshire, UK, 2000).

[19]

B.J. Winer, O.R. Brown, and K.M. Michels, Statistical Principles in Experimental Design, McGraw-Hill Series in Psychology, 3 rd ed., 1991.

[20]

C. Wohlin, and P. Runeson, "Defect Content Estimations from Review Data." in Proc. 20 th Int. Conf. on Software Engineering (1998), pp. 400-409.

Digital Library

[21]

C. Wohlin, P. Runeson, M. H~st, M.C. Ohlsson, B. Regnell, and A. Wessl~n. Experimentation in Software Engineering - An Introduction. The Kluwer International Series in Software Engineering, Kluwer Academic Publishers, 2000.

Digital Library

Cited By

Zhang QRong GZhang HBeecham SKitchenham BMacDonell S(2016)An empirical study on independence-driven data selection for improving capture-recapture estimationProceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering10.1145/2915970.2915991(1-10)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2915970.2915991
Kalinowski MTravassos G(2004)A Computational Framework for Supporting Software InspectionsProceedings of the 19th IEEE international conference on Automated software engineering10.5555/1025115.1025206(46-55)Online publication date: 20-Sep-2004
https://dl.acm.org/doi/10.5555/1025115.1025206
Padberg FRagg TSchoknecht R(2004)Using Machine Learning for Estimating the Defect Content After an InspectionIEEE Transactions on Software Engineering10.1109/TSE.2004.126573330:1(17-28)Online publication date: 1-Jan-2004
https://dl.acm.org/doi/10.1109/TSE.2004.1265733
Show More Cited By

Index Terms

Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Software management
      2. System management
        Quality assurance
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Investigating the Accuracy of Defect Estimation Models for Individuals and Teams Based on Inspection Data
ISESE '03: Proceedings of the 2003 International Symposium on Empirical Software Engineering

Defect content estimation approaches, based on datafrom inspection, estimate the total number of defects in adocument to evaluate the quality of the product and thedevelopment process. Objective estimation approachesrequire a high-quality measurement ...
Improving defect detection ability with adaptive software inspection checklists
Evaluation of inspectors' defect estimation accuracy for a requirements document after individual inspection
APSEC '00: Proceedings of the Seventh Asia-Pacific Software Engineering Conference

Project managers need timely feedback on the quality of development products to monitor and control project progress. Inspection is an effective method to identify defects and to measure product quality. Objective and subjective models can be used to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '01: Proceedings of the 23rd International Conference on Software Engineering

July 2001

844 pages

ISBN:0769510507

Chairman:
Hausi A. Müller
Univ. of Victoria, Canada

Copyright © Copyright (c) 2001 Institute of Electrical and Electronics Engineers, Inc. All rights reserved.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 July 2001

Check for updates

Qualifiers

Article

Conference

ICSE01

Sponsor:

SIGSOFT

ICSE01: 23rd International Conference on Software Engineering

May 12 - 19, 2001

Ontario, Toronto, Canada

Acceptance Rates

ICSE '01 Paper Acceptance Rate 47 of 268 submissions, 18%;

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
591
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang QRong GZhang HBeecham SKitchenham BMacDonell S(2016)An empirical study on independence-driven data selection for improving capture-recapture estimationProceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering10.1145/2915970.2915991(1-10)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2915970.2915991
Kalinowski MTravassos G(2004)A Computational Framework for Supporting Software InspectionsProceedings of the 19th IEEE international conference on Automated software engineering10.5555/1025115.1025206(46-55)Online publication date: 20-Sep-2004
https://dl.acm.org/doi/10.5555/1025115.1025206
Padberg FRagg TSchoknecht R(2004)Using Machine Learning for Estimating the Defect Content After an InspectionIEEE Transactions on Software Engineering10.1109/TSE.2004.126573330:1(17-28)Online publication date: 1-Jan-2004
https://dl.acm.org/doi/10.1109/TSE.2004.1265733
Ragg TPadberg FSchoknecht R(2002)Applying Machine Learning to Solve an Estimation Problem in Software InspectionsProceedings of the International Conference on Artificial Neural Networks10.5555/646259.684311(516-521)Online publication date: 28-Aug-2002
https://dl.acm.org/doi/10.5555/646259.684311
Padberg FTracz WMagee JYoung M(2002)Empirical interval estimates for the defect content after an inspectionProceedings of the 24th International Conference on Software Engineering10.1145/581339.581350(58-68)Online publication date: 19-May-2002
https://dl.acm.org/doi/10.1145/581339.581350
Biffl SGutjahr W(2002)Using a Reliability Growth Model to Control Software InspectionEmpirical Software Engineering10.1023/A:10163962324487:3(257-284)Online publication date: 1-Sep-2002
https://dl.acm.org/doi/10.1023/A%3A1016396232448

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents