Abstract
Logic coverage criteria have been widely used in the testing of safety-critical software. In the past few years, fault-based logic coverage criteria have been studied intensively both in theory and in practice. However, there is a lack of authentic evidence of the comparison of fault-based logic coverage criteria with other logic coverage criteria, such as branch coverage and modified condition/decision coverage (MC/DC). In this paper, we present a comprehensive empirical study that compares these logic coverage criteria on test case prioritization, which is currently a hot topic in software testing. Several useful conclusions are drawn from our research: (1) Fine-grained coverage criteria are always more effective and efficient. (2) The effectiveness of fault-based logic coverage criteria is not significantly different from that of MC/DC in terms of statistics, but the former is more stable. (3) A random strategy is more effective than branch coverage if a certain number of test cases are redundant.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Weyuker E, Goradia T, Singh A. Automatically generating test data from a Boolean specification. IEEE Trans Softw Eng, 1994, 20: 353–363
Chen T Y, Lau M F, Yu Y T. Mumcut: A fault-based strategy for testing Boolean specifications. In: Proceedings of the Sixth Asia Pacific Software Engineering Conference (APSEC’99), Takamatsu, 1999. 606–613
Kaminski G, Williams G, Ammann P. Reconciling perspectives of software logic testing. Softw Test Verif Reliab, 2008, 18: 149–188
Chen Z, Chen T Y, Xu B. A revisit of fault class hierarchies in general Boolean specifications. ACM Trans Softw Eng Methodol, 2011, 20: 13
Cornett S. Code coverage analysis. Bullseye Testing Technology. http://www.bullseye.com/coverage.html
Chilenski J J, Miller S. Applicability of modified condition/decision coverage to software testing. Softw Eng J, 1994, 9: 193–200
Yu Y T, Lau M F. A comparison of MC/DC, MUMCUT and several other coverage criteria for logical decisions. J Syst Softw, 2006, 79: 577–590
Ammann P, Offutt J, Huang H. Coverage criteria for logical expressions. In: Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03). Washington DC: IEEE, 2003. 99–107
RCTA. Do-178B: Software considerations in airborne systems and equipment certification. http://www.dcs.gla.ac.uk/~johnson/teaching/safety/reports/schad.html
Lau M F, Yu Y T. An extended fault class hierarchy for specification-based testing. ACM Trans Softw Eng Methodol, 2005, 14: 247–276
Kuhn D R. Fault classes and error detection capability of specification-based testing. ACM Trans Softw Eng Methodol, 1999, 8: 411–424
Yu Y T, Lau M F. Comparing several coverage criteria for detecting faults in logical decisions. In: Proceedings of the 4th International Conference on Quality Software (QSIC’04), Braunschweig, 2004. 14–21
Chen T Y, Lau M F. Test case selection strategies based on Boolean specifications. Softw Test Verif Reliab, 2001, 11: 165–180
Foster K A. Sensitive test data for logic expressions. ACM SIGSOFT Softw Eng Notes, 1984, 9: 120–125
Daran M, Thévenod-Fosse P. Software error analysis: a real case study involving real faults and mutations. ACM SIGSOFT Softw Eng Notes, 1996, 21: 158–171
Elbaum S, Malishevsky A G, Rothermel G. Test case prioritization: A family of empirical studies. IEEE Trans Softw Eng, 2002, 28: 159–182
Wong W E, Horgan J R, London S, et al. A study of effective regression testing in practice. In: Proceedings of the 8th International Symposium On Software Reliability Engineering (ISSRE’97), Albuquerque, 1997. 264–274
Jones J A, Harrold M J. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans Softw Eng, 2003, 29: 195–209
Chen Z, Xu B, Ding D. The complexity results of variable minimal formulas. Chin Sci Bull, 2010, 55: 1957–1960
Kapoor K, Bowen J. Experimental evaluation of the variation in effectiveness for DC, FPC and MC/DC test criteria. In: Proceedings of the 2nd International Symposium on Empirical Software Engineering (ISESE’03), Rome, 2003. 185–194
Singh R K, Chandra P, Singh Y. An evaluation of Boolean expression testing techniques. ACM SIGSOFT Softw Eng Notes, 2006, 31: 1–6
Gupta A, Jalote P. An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing. Int Int J Softw Tools Technol Transf, 2008, 10: 145–160
Rothermel G, Untch R H, Chu C, et al. Test case prioritization: An empirical study. In: Proceedings of the 7th IEEE International Conference on Software Maintenance (ICSM’99), Oxford, 1999. 179–188
Wang Z, Chen Z, Chen T Y, et al. Fault class prioritization in Boolean expressions. In: Proceedings of the 27th Annual ACM Symposium on Applied Computing (SAC’12). New York: ACM, 2012. 1191–1196
Rothermel G, Untch R H, Chu C, et al. Prioritizing test cases for regression testing. IEEE Trans Softw Eng, 2001, 27: 929–948
Li Z, Harman M, Hierons R M. Search algorithms for regression test case prioritization. IEEE Trans Softw Eng, 2007, 33: 225–237
Strecker J, Memon A M. Accounting for defect characteristics in evaluations of testing techniques. ACM Trans Softw Eng Methodol, 2012, 21: 17
Team C. CodeCover homepage. http://codecover.org/
Arcuri A, Briand L. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd International Conference on Software Engineering (ICSE’11), Honolulu, 2011. 1–10
Staats M, Whalen M W, Heimdahl M P E. Programs, tests, and oracles: the foundations of testing revisited. In: Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). New York: ACM, 2011. 391–400
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Fang, C., Chen, Z. & Xu, B. Comparing logic coverage criteria on test case prioritization. Sci. China Inf. Sci. 55, 2826–2840 (2012). https://doi.org/10.1007/s11432-012-4746-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11432-012-4746-9