Article

Assisted Coverage Closure

Authors:

Pascal Kesseli,

Philippa Ryan Conmy,

Daniel Kroening,

Peter Schrammel,

Michael TautschnigAuthors Info & Claims

NFM 2016: Proceedings of the 8th International Symposium on NASA Formal Methods - Volume 9690

Pages 49 - 64

https://doi.org/10.1007/978-3-319-40648-0_5

Published: 07 June 2016 Publication History

Abstract

Malfunction of safety-critical systems may cause damage to people and the environment. Software within those systems is rigorously designed and verified according to domain specific guidance, such as ISO26262 for automotive safety. This paper describes academic and industrial co-operation inï źtool development to support one of the most stringent of the requirements -- achieving full code coverage in requirements-driven testing. We present a verification workflow supported by a tool that integrates the coverage measurement tool RapiCover with the test-vector generator FShell. The tool assists closing the coverage gap by providing the engineer with test vectors that help in debugging coverage-related code quality issues and creating new test cases, as well as justifying the presence of unreachable parts of the code in order to finally achieve full effective coverage according to the required criteria. We illustrate the tool's practical utility on automotive industry benchmarks. It generates 8$$\times $$ more MC/DC coverage than random search.

References

[1]

ISO26262 road vehicles --- functional safety, Part 6: Product development at the software level, Annex B: Model-based development 2011

[2]

Biere, A., Cimatti, A., Clarke, E., Zhu, Y.: Symbolic model checking without BDDs. In: Cleaveland, W.R. ed. TACAS 1999. LNCS, vol. 1579, p. 193. Springer, Heidelberg 1999

Digital Library

[3]

Clarke, E., Kroening, D., Lerda, F.: A tool for checking ANSI-C programs. In: Jensen, K., Podelski, A. eds. TACAS 2004. LNCS, vol. 2988, pp. 168---176. Springer, Heidelberg 2004

[4]

Dupuy, A., Leveson, N.: An empirical evaluation of the MC/DC coverage criterion on the HETE-2 satellite software. In: Digital Avionics Systems Conference, vol. 1, pp. 1B6/1---1B6/7 2000

[5]

Fraser, G., Wotawa, F., Ammann, P.: Testing with model checkers: a survey. Softw. Test., Verification Reliab. 193, 215---261 2009

Digital Library

[6]

Ghani, K., Clark, J.A.: Automatic test data generation for multiple condition and MCDC coverage. In: ICSEA, pp. 152---157 2009

Digital Library

[7]

Holzer, A., Schallhart, C., Tautschnig, M., Veith, H.: FShell: systematic test case generation for dynamic analysis and measurement. In: Gupta, A., Malik, S. eds. CAV 2008. LNCS, vol. 5123, pp. 209---213. Springer, Heidelberg 2008

Digital Library

[8]

Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. Trans. Software Eng. 375, 649---678 2011

Digital Library

[9]

Kandl, S., Kirner, R.: Error detection rate of MC/DC for a case study from the automotive domain. In: Min, S.L., Pettit, R., Puschner, P., Ungerer, T. eds. SEUS 2010. LNCS, vol. 6399, pp. 131---142. Springer, Heidelberg 2010

Digital Library

[10]

Kroening, D., Strichman, O.: Efficient computation of recurrence diameters. In: Zuck, L.D., Attie, P.C., Cortesi, A., Mukhopadhyay, S. eds. VMCAI 2003. LNCS, vol. 2575, pp. 298---309. Springer, Heidelberg 2002

Digital Library

[11]

Rierson, L.: Developing Safety-Critical Software: A Practical Guide for Aviation Software and DO-178C Compliance. CRC Press, Boca Raton 2013. Chapter 14.3 Potential Risks of Model-Based Development and Verification

[12]

Schrammel, P., Melham, T., Kroening, D.: Chaining test cases for reactive system testing. In: Yenigün, H., Yilmaz, C., Ulrich, A. eds. ICTSS 2013. LNCS, vol. 8254, pp. 133---148. Springer, Heidelberg 2013

[13]

Sheeran, M., Singh, S., Stålmarck, G.: Checking safety properties using induction and a SAT-solver. In: Johnson, S.D., Hunt Jr., W.A. eds. FMCAD 2000. LNCS, vol. 1954, pp. 108---125. Springer, Heidelberg 2000

Digital Library

[14]

Tallam, S., Gupta, N.: A concept analysis inspired greedy algorithm for test suite minimization. In: PASTE, pp. 35---42 2005

Digital Library

[15]

Zamli, K.Z., Al-Sewari, A.A., Hassin, M.H.M.: On test case generation satisfying the MC/DC criterion. Int. J. Adv. Soft Comput. Appl. 53 2013

Cited By

Güdemann MSchrammel PHong JLanperne MPark JCerny TShahriar H(2023)BlueCov: Integrating Test Coverage and Model Checking with JBMCProceedings of the 38th ACM/SIGAPP Symposium on Applied Computing10.1145/3555776.3577829(1695-1697)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3555776.3577829

Recommendations

Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

Many educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage ...
State coverage: a structural test adequacy criterion for behavior checking
ESEC-FSE '07: Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering

We propose a new language-independent, structural test adequacy criterion called state coverage. State coverage measures whether unit-level tests check the outputs and sideeffects of a program.

State coverage differs in several respects from existing ...
State coverage: a structural test adequacy criterion for behavior checking
ESEC-FSE companion '07: The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papers

We propose a new language-independent, structural test adequacy criterion called state coverage. State coverage measures whether unit-level tests check the outputs and side effects of a program.

State coverage differs in several respects from existing ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

NFM 2016: Proceedings of the 8th International Symposium on NASA Formal Methods - Volume 9690

June 2016

376 pages

ISBN:9783319406473

Editors:
Sanjai Rayadurgam
University of Minnesota, Minneapolis, USA
,
Oksana Tkachuk
NASA Ames Research Center, Moffett Field, USA

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 07 June 2016

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Güdemann MSchrammel PHong JLanperne MPark JCerny TShahriar H(2023)BlueCov: Integrating Test Coverage and Model Checking with JBMCProceedings of the 38th ACM/SIGAPP Symposium on Applied Computing10.1145/3555776.3577829(1695-1697)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3555776.3577829

View Options

View options

Figures

Tables

Media

View Table of Conten