research-article

Open access

Concolic testing of the multi-sector read operation for flash storage platform software

Authors:

Yunja ChoiAuthors Info & Claims

Formal Aspects of Computing, Volume 24, Issue 3

Pages 355 - 374

https://doi.org/10.1007/s00165-011-0200-9

Published: 01 May 2012 Publication History

Abstract

In today’s information society, flash memory has become a virtually indispensable component, particularly for mobile devices. In order for mobile devices to operate successfully, it is essential that flash memory be controlled correctly through flash storage platform software such as the file system, flash translation layer, and low-level device drivers. However, as is typical for embedded software, conventional testing methods often fail to detect hidden flaws in the software due to the difficulty of creating effective test cases. As a different approach, model checking techniques guarantee a complete analysis, but only on a limited scale. In this paper, we describe an empirical study wherein a concolic testing method is applied to the multi-sector read operation for flash storage platform software. This method combines a concrete dynamic execution and a symbolic execution to automatically generate test cases for full path coverage. Through the experiments, we analyze the advantages and weaknesses of the concolic testing approach on the flash storage platform software.

References

References

[1]

Andrews JH, Briand LC, Labiche Y (2005) Is mutation an appropriate tool for testing experiments? In International conference on software engineering, St.Louis, Missouri, USA, 15–21 May 2005, pp 402–411

[2]

Amazon.com, Inc. Amazon Elastic Compute Cloud (Amazon EC2). http://aws.amazon.com/ec2/

[3]

Ammann P and Offutt J Introduction to software testing 2008 Cambridge Cambridge Press

Digital Library

[4]

Botella B, Delahaye M, Hong-Tuan-Ha S, Kosmatov N, Mouy P, Roger M, Williams N (2009) Automating structural testing of c programs: experience with pathcrawler. In: Automation of software test. Vancouver, BC, Canada, 18–19 May 2009

[5]

Barrett C, de Moura L, Stump A (2005) SMT-COMP: satisfiability modulo theories competition. In: Computer aided verification, Scotland, UK, 6–10 July 2005, pp 20–23

[6]

Butterfield A, Freitas L, and Woodcock J Mechanising a formal model of flash memory Sci Comput Program 2009 74 4 219-237

Digital Library

[7]

Berezin S, Ganesh V, Dill DL (2003) An online proof-producing decision procedure for mixed integer linear arithmetic. In: Tools and algorithms for the construction and analysis of systems, Warsaw, Poland, 7–11 April 2003, pp 521–536

[8]

Burnim J, Sen K (2008) Heuristics for scalable dynamic test generation. Technical report UCB/EECS-2008-123, EECS Department, University of California, Berkeley, Sep 2008

[9]

Barrett C, Stump A, Tinelli C (2010) The satisfiability modulo theories library (SMT-LIB). http://www.SMT-LIB.org

[10]

Burnim J (2008) CREST—automatic test generation tool for C http://code.google.com/p/crest/

[11]

Cadar C, Dunbar D, Engler D (2008) KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Operating system design and implementation, San Diego, CA, USA, 8–10 Dec 2008, pp 209–224

[12]

Clarke E, Kroening D, Lerda F (2004) A tool for checking ANSI-C programs. In: Tools and algorithms for the construction and analysis of systems, Barcelona, Spain, 29 March–2 April 2004, pp 168–176

[13]

Do H, Elbaum S, and Rothermel G Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact Empir Softw Eng J 2005 10 4 405-435

Digital Library

[14]

Dutertre B, Moura L (2006) A fast linear-arithmetic solver for DPLL(T). In: Computer aided verification, Seattle, WA, USA, 17–20 Aug 2006, pp 81–94

[15]

Do H, Rothermel G, Elbaum S (2004) Infrastructure support for controlled experimentation with software testing and regression testing techniques. Technical report 04-06-01, Oregon State Univ, Jan 2004

[16]

Emmi M, Majumdar R, Sen K (2007) Dynamic test input generation for database applications. In: International symposium on software testing and analysis, London, UK, 9–12 July 2007, pp 151–161

[17]

Een N, Sorensson N (2003) An extensible sat-solver. In: SAT, S. Margherita Ligure, Italy, 5–8 May 2003, pp 502–518

[18]

Ferreira MA, Silva SS, Oliveira JN (2008) Verifying Intel flash file system core specification. In: VDM-overture workshop, Turku, Finland, 26 May 2008, pp 54–71

[19]

Godefroid P, Kiezun A, Levin MY (2008) Grammar-based whitebox fuzzing. In: Programming language design and implementation, Tucson, Arizona, USA, 7–13 June 2008, pp 206–215

[20]

Godefroid P, Klarlund N, Sen K (2005) DART: directed automated random testing. In: Programming language design and implementation Chicago, IL, USA, 12–15 June 2005, pp 213–223

[21]

Godefroid P, Levin MY, Molnar D (2008) Automated whitebox fuzz testing. In: Network and distributed systems security, San Diego, CA, USA, 8–11 Feb 2008

[22]

Holzmann G The spin model checker 2003 New York Wiley

Digital Library

[23]

Jayaraman K, Harvison D, Ganesh V, Kiezun A (2009) jFuzz: a concolic whitebox fuzzer for Java. In: NASA formal methods symposium. Moffett Field, CA, USA, 6–8 April 2009, pp 121–125

[24]

Kim M, Choi Y, Kim Y, Kim H (2008) Formal verification of a flash memory device driver—an experience report. In: Spin workshop, LA, USA, 10–12 Aug 2008, pp 144–159

[25]

King JC Symbolic execution and program testing Commun ACM 1976 19 7 385-394

Digital Library

[26]

Kang E, Jackson D (2008) Formal modeling and analysis of a flash filesystem in alloy. In: Abstract state machines, B and Z. London, UK, 16–18 Sep 2008, pages 294–308

[27]

Kim M, Kim Y (2009) Concolic testing of the multi-sector read operation for flash memory file system. In: Brazilian symposium on formal methods (SBMF), Gramado, Brazil, 19–21 Aug 2009, pp 251–265

[28]

Kim Y, Kim M (2011) SCORE: a scalable concolic testing tool for reliable embedded software. In: European software engineering conference/foundations of software engineering, Szeged, Hungary, 5–9 Sep 2011

[29]

Kim M, Kim Y, Choi Y, Kim H (2008) Pre-testing flash device driver through model checking techniques. In: International conference on software testing, verification and validation, Lillehammer, Norway, 9–11 Apr 2008, pp 475–484

[30]

Kim Y, Kim M, Dang N (2010) Scalable distributed concolic testing: a case study on a flash storage platform. In: International colloquium on theoretical aspects of computing, Natal, Brazil, 1–3 Sep 2010, pp 199–213

[31]

Kim M, Kim Y, and Kim H Comparative study on software model checkers as unit testing tools: an industrial case study IEEE Trans Softw Eng (TSE) 2011 37 2 146-160

Digital Library

[32]

Lattner C, Adve V (2004) LLVM: a compilation framework for lifelong program analysis & transformation. In: International symposium on code generation and optimization, Palo Alto, CA, USA, 21–24 Mar 2004, pp 75–88

[33]

Lakhotia K, Harman M, McMinn P (2008) Handling dynamic data structures in search based testing. In: Genetic and evolutionary computation, Atlanta, GA, USA, 12–16 July 2008, pp 1759–1766

[34]

Lakhotia K, McMinn P, Harman M (2009) Automated test data generation for coverage: haven’t we solved this problem yet? In: Testing: academic & industrial conference practice and research techniques, Windsor, UK, 4–6 Sep 2009, pp 95–104

[35]

Moura L, Bjorner N (2008) Z3: an efficient SMT solver. In: Tools and algorithms for the construction and analysis of systems, Budapest, Hungary, 29 March–6 April 2008, pp 337–340

[36]

Majumdar R, Xu R (2007) Directed test generation with symbolic grammars. In: Automated software engineering, Atlanta, GA, USA, 5–9 Nov 2007, pp 134–143

[37]

Marri M, Xie T, Tillmann N, Halleux J, Schulte W (2009) An empirical study of testing file-system-dependent software with mock objects. In: Automation of software test, Vancouver, BC, Canada, 18–19 May 2009, pp 149–153

[38]

Nethercote N, Seward J (2007) Valgrind: a framework for heavyweight dynamic binary instrumentation. In: Programming language design and implementation, San Diego, California, USA, 11–13 June 2007, pp 89–100

[39]

Offutt AJ Investigations of the software testing coupling effect ACM Trans Softw Eng Methodol 1992 1 1 3-18

Digital Library

[40]

Offutt AJ, Lee A, Rothermel G, Untch RH, and Zapf C An experimental determination of sufficient mutation operators ACM Trans Softw Eng Methodol 1996 5 2 99-118

Digital Library

[41]

Samsung OneNAND fusion memory. http://www.samsung.com/global/business/semiconductor/products/fusionmemory/Products_OneNAND.html

[42]

Pasareanu C and Visser W A survey of new trends in symbolic execution for software testing and analysis Softw Tools Technol Transf 2009 11 4 339-353

[43]

QF_LIA benchmark: nec-smt/large/checkpass/prp-19-47.smt, SMT-COMP 2009. http://www.smtexec.org/exec/benchmarkResults.php?jobs=529&benchmark=1087660

[44]

Sen K, Agha G (2006) CUTE and jCUTE: concolic unit testing and explicit path model-checking tools. In: Computer aided verification, Seattle, WA, USA, 17–20 Aug 2006, pp 419–423

[45]

Sen K, Marinov D, Agha G (2005) CUTE: a concolic unit testing engine for C. In: European software engineering conference/foundations of software engineering, Lisbon, Portugal, 5–9 Sep 2005, pp 263–272

[46]

Tillmann N, Schulte W (2005) Parameterized unit tests. In: European software engineering conference/foundations of software engineering, Lisbon, Portugal, 5–9 Sep 2005, pp 253–262

[47]

Visser W, Havelund K, Brat G, Park S (2000) Model checking programs. In: Automated software engineering, September 2000

[48]

Visser W, Pasareanu CS, Khurshid S (2004) Test input generation with Java PathFinder. In: International symposium on software testing and analysis, Boston MA, USA, 11–14 July 2004, pp 97–107

[49]

Williams N, Marre B, Mouy P, Roger M (2005) Pathcrawler: automatic generation of path tests by combining static and dynamic analysis. In: European dependable computing conference, Budapest, Hungary, 20–22 Apr 2005, pp 281–292

Cited By

Yuan JAoki TGuo X(2022)Comprehensive evaluation of file systems robustness with SPIN model checkingSoftware Testing, Verification and Reliability10.1002/stvr.182832:6Online publication date: 20-Jul-2022
https://doi.org/10.1002/stvr.1828
Mayeda MAndrews A(2021)Evaluating software testing techniques: A systematic mapping study10.1016/bs.adcom.2021.01.002(41-114)Online publication date: 2021
https://doi.org/10.1016/bs.adcom.2021.01.002
Fan MWang WYu AMeng D(2021)TDCT: Target-Driven Concolic Testing Using Extended Units by Calculating Function RelevanceAlgorithms and Architectures for Parallel Processing10.1007/978-3-030-95391-1_13(196-213)Online publication date: 3-Dec-2021
https://dl.acm.org/doi/10.1007/978-3-030-95391-1_13
Show More Cited By

Index Terms

Concolic testing of the multi-sector read operation for flash storage platform software
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Formal software verification
      2. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Software functional properties
      1. Formal methods
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning

Index terms have been assigned to the content through auto-classification.

Recommendations

Concolic Testing of the Multi-sector Read Operation for Flash Memory File System
Formal Methods: Foundations and Applications

In today's information society, flash memory has become a virtually indispensable component, particularly for mobile devices. In order for mobile devices to operate successfully, it is essential that flash memory be controlled correctly through file ...
Read leveling for flash storage systems
SYSTOR '15: Proceedings of the 8th ACM International Systems and Storage Conference

Due to its several attractive benefits such as shock resistance, energy efficiency, and space-efficient form factor, flash memory is now applied to a wide range of electronics. Typically, since write requests are harmful to the health of flash memory, ...
Performance Trade-Offs in Using NVRAM Write Buffer for Flash Memory-Based Storage Devices

While NAND flash memory is used in a variety of end-user devices, it has a few disadvantages, such as asymmetric speed of read and write operations, inability to in-place updates, among others. To overcome these problems, various flash-aware strategies ...

Comments

Information & Contributors

Information

Published In

cover image Formal Aspects of Computing

Formal Aspects of Computing Volume 24, Issue 3

May 2012

111 pages

ISSN:0934-5043

EISSN:1433-299X

Issue’s Table of Contents

© British Computer Society 2011.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 May 2012

Accepted: 01 November 2010

Revision received: 23 August 2010

Received: 10 March 2010

Published in FAC Volume 24, Issue 3

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
65
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)9

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yuan JAoki TGuo X(2022)Comprehensive evaluation of file systems robustness with SPIN model checkingSoftware Testing, Verification and Reliability10.1002/stvr.182832:6Online publication date: 20-Jul-2022
https://doi.org/10.1002/stvr.1828
Mayeda MAndrews A(2021)Evaluating software testing techniques: A systematic mapping study10.1016/bs.adcom.2021.01.002(41-114)Online publication date: 2021
https://doi.org/10.1016/bs.adcom.2021.01.002
Fan MWang WYu AMeng D(2021)TDCT: Target-Driven Concolic Testing Using Extended Units by Calculating Function RelevanceAlgorithms and Architectures for Parallel Processing10.1007/978-3-030-95391-1_13(196-213)Online publication date: 3-Dec-2021
https://dl.acm.org/doi/10.1007/978-3-030-95391-1_13
Gleirscher MFoster SWoodcock J(2019)New Opportunities for Integrated Formal MethodsACM Computing Surveys10.1145/335723152:6(1-36)Online publication date: 16-Oct-2019
https://dl.acm.org/doi/10.1145/3357231
Kim YLee DBaek JKim MSharp HWhalen M(2019)Concolic testing for high test coverage and reduced human effort in automotive industryProceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP.2019.00024(151-160)Online publication date: 27-May-2019
https://dl.acm.org/doi/10.1109/ICSE-SEIP.2019.00024
Godboley SDutta AMohapatra DMall R(2018)GECOJAPComputer Standards & Interfaces10.1016/j.csi.2017.04.00355:C(27-46)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.csi.2017.04.003
Godboley SDutta AMohapatra DMall R(2017)J3 ModelComputer Standards & Interfaces10.1016/j.csi.2016.09.00650:C(1-17)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1016/j.csi.2016.09.006
Godboley SDutta AMohapatra DMall R(2017)Green-J $$^{3}$$ 3 Model: a novel approach to measure energy consumption of modified condition/decision coverage using concolic testingCSI Transactions on ICT10.1007/s40012-017-0157-95:3(217-233)Online publication date: 7-Feb-2017
https://doi.org/10.1007/s40012-017-0157-9
Godboley SMohapatra DDas AMall R(2017)An improved distributed concolic testing approachSoftware10.1002/spe.240547:2(311-342)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1002/spe.2405
Godboley SPanda SDutta AMohapatra D(2016)An Automated Analysis of the Branch Coverage and Energy Consumption Using Concolic TestingArabian Journal for Science and Engineering10.1007/s13369-016-2284-242:2(619-637)Online publication date: 27-Aug-2016
https://doi.org/10.1007/s13369-016-2284-2
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents