Abstract
Many evolving mission‐critical systems must have high software reliability. However, it is often difficult to identify fault‐prone modules early enough in a development cycle to guide software enhancement efforts effectively and efficiently. Software quality models can yield timely predictions of membership in the fault‐prone class on a module‐by‐module basis, enabling one to target enhancement techniques. However, it is an open empirical question, “Can a software quality model remain useful over several releases?” Most prior software quality studies have examined only one release of a system, evaluating the model with modules from the same release. We conducted a case study of a large legacy telecommunications system where measurements on one software release were used to build models, and three subsequent releases of the same system were used to evaluate model accuracy. This is a realistic assessment of model accuracy, closely simulating actual use of a software quality model. A module was considered fault‐prone if any of its faults were discovered by customers. These faults are extremely expensive due to consequent loss of service and emergency repair efforts. We found that the model maintained useful accuracy over several releases. These findings are initial empirical evidence that software quality models can remain useful as a system is maintained by a stable software development process.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Basili, V.R., L.C. Briand, and W. Melo (1996), “A Validation of Object-Oriented Design Metrics as Quality Indicators,” IEEE Transactions on Software Engineering 22, 10, 751–761.
Biyani, S. and P. Santhanam (1998), “Exploring Defect Data from Development and Customer Usage on Software Modules over Multiple releases,” In Proceedings the Ninth International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Los Alamitos, CA, pp. 316–320.
Briand, L.C. and V.R. Basili (1992), “A Classification Procedure for the Effective Management of Changes during the Maintenance Process,” In Proceedings of the International Conference on Software Maintenance, IEEE Computer Society Press, Los Alamitos, CA, pp. 328–336.
Briand, L.C., V.R. Basili, and C.J. Hetmanski (1993), “Developing Interpretable Models with Optimized Set Reduction for Identifying High-Risk Software Components,” IEEE Transactions on Software Engineering 19, 11, 1028–1044.
Coleman, D., B. Lowther, and P. Oman (1995), “The Application of Software Maintainability Models in Industrial Software,” Journal of Systems and Software 29, 1, 3–16.
Ebert, C. (1996), “Classification Techniques for Metric-Based Software Development,” Software Quality Journal 5, 4, 255–272.
Fayyad, U.M. (1996), “Data Mining and Knowledge Discovery: Making Sense Out of Data,” IEEE Expert 11, 4, 20–25.
Hosmer, D.W., Jr. and S. Lemeshow (1989), “Applied Logistic Regression,” Wiley, New York.
Hudepohl, J.P., S.J. Aud, T.M. Khoshgoftaar, E.B. Allen, and J. Mayrand (1996), “EMERALD: Software Metrics and Models on the Desktop,” IEEE Software 13, 5, 56–60.
Jones, W.D., J.P. Hudepohl, T.M. Khoshgoftaar, and E.B. Allen (1999), “Application of a Usage Profile in Software QualityModels,” In Proceedings of the Third European Conference on Software Maintenance and Reengineering, IEEE Computer Society Press, Los Alamitos, CA, pp. 148–157.
Khoshgoftaar, T.M. and E.B. Allen (1999), “A Practical Classification Rule for Software Quality Models,” IEEE Transactions on Reliability 48.
Khoshgoftaar, T.M., E.B. Allen, K.S. Kalaichelvan, and N. Goel (1996), “Early Quality Prediction: A Case Study in Telecommunications,” IEEE Software 13, 1, 65–71.
Khoshgoftaar, T.M., E.B. Allen, R. Halstead, G.P. Trio, and R. Flass (1998a), “Process Measures for Predicting Software Quality,” Computer 31, 4, 66–72.
Khoshgoftaar, T.M., E.B. Allen, W.D. Jones, and J.P. Hudepohl (1998b), “Return on Investment of Software Quality Models,” In Proceedings 1998 IEEE Workshop on Application-Specific Software Engineering and Technology, IEEE Computer Society Press, Los Alamitos, CA, pp. 145–150.
Khoshgoftaar, T.M., E.B. Allen, A. Naik, W.D. Jones, and J.P. Hudepohl (1998c), “Using Classification Trees for Software Quality Models: Lessons Learned,” In Proceedings of the Third IEEE International High-Assurance Systems Engineering Symposium, IEEE Computer Society Press, Los Alamitos, CA, pp. 82–89.
Khoshgoftaar, T.M., E.B. Allen, W.D. Jones, and J.P. Hudepohl (1999), “Which Software Modules Have Faults that Will Be Discovered by Customers?” Journal of Software Maintenance: Research and Practice 11, 1, 1–18.
Khoshgoftaar, T.M. and D.L. Lanning (1995), “A Neural Network Approach for Early Detection of Program Modules having High Risk in the Maintenance Phase,” Journal of Systems and Software 29, 1, 85–91.
Mayrand, J. and F. Coallier (1996), “System Acquisition Based on Software Product Assessment,” In Proceedings of the Eighteenth International Conference on Software Engineering, IEEE Computer Society Press, Los Alamitos, CA, pp. 210–219.
Munson, J.C. and T.M. Khoshgoftaar (1992), “The Detection of Fault-Prone Programs,” IEEE Transactions on Software Engineering 18, 5, 423–433.
Myers, R.H. (1990), Classical and Modern Regression with Applications, Duxbury Series, PWS-KENT Publishing, Boston.
Schneidewind, N.F. (1995), “Software Metrics Validation: Space Shuttle Flight Software Example,” Annals of Software Engineering 1, 287–309.
Schneidewind, N.F. (1998), “An Integrated Process and Product Model,” In Proceedings Fifth International Software Metrics Symposium, IEEE Computer Society Press, Los Alamitos, CA, pp. 224–234.
Seber, G.A.F. (1984), Multivariate Observations, Wiley, New York.
Selby, R.W. and A.A. Porter (1988), “Learning from Examples: Generation and Evaluation of Decision Trees for Software Resource Analysis,” IEEE Transactions on Software Engineering 14, 12, 1743–1756.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Khoshgoftaar, T.M., Allen, E.B., Jones, W.D. et al. Accuracy of software quality models over multiple releases. Annals of Software Engineering 9, 103–116 (2000). https://doi.org/10.1023/A:1018972607783
Issue Date:
DOI: https://doi.org/10.1023/A:1018972607783