research-article

Automatically testing implementations of numerical abstract domains

Authors:

Alexandra Bugariu,

Valentin Wüstholz,

Maria Christakis,

Peter MüllerAuthors Info & Claims

ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

Pages 768 - 778

https://doi.org/10.1145/3238147.3240464

Published: 03 September 2018 Publication History

Abstract

Static program analyses are routinely applied as the basis of code optimizations and to detect safety and security issues in software systems. For their results to be reliable, static analyses should be sound (i.e., should not produce false negatives) and precise (i.e., should report a low number of false positives). Even though it is possible to prove properties of the design of a static analysis, ensuring soundness and precision for its implementation is challenging. Complex algorithms and sophisticated optimizations make static analyzers difficult to implement and test.

In this paper, we present an automatic technique to test, among other properties, the soundness and precision of abstract domains, the core of all static analyzers based on abstract interpretation. In order to cover a wide range of test data and input states, we construct inputs by applying sequences of abstract-domain operations to representative domain elements, and vary the operations through gray-box fuzzing. We use mathematical properties of abstract domains as test oracles. Our experimental evaluation demonstrates the effectiveness of our approach. We detected several previously unknown soundness and precision errors in widely-used abstract domains. Our experiments also show that our approach is more effective than dynamic symbolic execution and than fuzzing the test inputs directly.

References

[1]

{n. d.}. The APRON Library Documentation. http://apron.cri.ensmp.fr/library/0. 9.10/apron.pdf.

[2]

{n. d.}. The Coq Proof Assistant. https://coq.inria.fr.

[3]

{n. d.}. ELINA Artifact (POPL 2017). https://www.sri.inf.ethz.ch/optpoly.php.

[4]

{n. d.}. ELINA Artifact (POPL 2018). https://www.sri.inf.ethz.ch/popl18-paper251. php.

[5]

{n. d.}. KLEE Tutorial. http://klee.github.io/tutorials/testing-regex/.

[6]

{n. d.}. LibFuzzer—A Library for Coverage-Guided Fuzz Testing. https://llvm. org/docs/LibFuzzer.html.

[7]

{n. d.}. Technical “Whitepaper” for AFL. http://lcamtuf.coredump.cx/afl/ technical_details.txt.

[8]

Esben Sparre Andreasen, Anders Møller, and Benjamin Barslev Nielsen. 2017. Systematic Approaches for Increasing Soundness and Precision of Static Analyzers. In SOAP. ACM, 31–36.

Digital Library

[9]

Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. TSE 41, 5 (2015), 507–525.

Digital Library

[10]

Dirk Beyer, Thomas A. Henzinger, M. Erkan Keremoglu, and Philipp Wendler. 2012. Conditional Model Checking: A Technique to Pass Information between Verifiers. In FSE. ACM, 57–67.

Digital Library

[11]

Dirk Beyer and M. Erkan Keremoglu. 2011. CPAchecker: A Tool for Configurable Software Verification. In CAV (LNCS), Vol. 6806. Springer, 184–190.

Digital Library

[12]

Bruno Blanchet, Patrick Cousot, Radhia Cousot, Jérome Feret, Laurent Mauborgne, Antoine Miné, David Monniaux, and Xavier Rival. 2003. A Static Analyzer for Large Safety-critical Software. In PLDI. ACM, 196–207.

Digital Library

[13]

Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In OSDI. USENIX, 209–224.

Digital Library

[14]

Cristian Cadar and Dawson R. Engler. 2005. Execution Generated Test Cases: How to Make Systems Code Crash Itself. In SPIN (LNCS), Vol. 3639. Springer, 2–23.

Digital Library

[15]

Maria Christakis, Peter Müller, and Valentin Wüstholz. 2012. Collaborative Verification and Testing with Explicit Assumptions. In FM (LNCS), Vol. 7436.

[16]

Springer, 132–146.

[17]

Maria Christakis, Peter Müller, and Valentin Wüstholz. 2015. An Experimental Evaluation of Deliberate Unsoundness in a Static Program Analyzer. In VMCAI (LNCS), Vol. 8931. Springer, 336–354.

Digital Library

[18]

Maria Christakis and Valentin Wüstholz. 2016. Bounded Abstract Interpretation. In SAS (LNCS), Vol. 9837. Springer, 105–125.

[19]

Koen Claessen and John Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In ICFP. ACM, 268–279.

Digital Library

[20]

Patrick Cousot and Radhia Cousot. 1976. Static Determination of Dynamic Properties of Programs. In ISOP. Dunod, 106–130.

[21]

Patrick Cousot and Radhia Cousot. 1977. Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In POPL. ACM, 238–252.

Digital Library

[22]

Patrick Cousot and Radhia Cousot. 1979. Systematic Design of Program Analysis Frameworks. In POPL. ACM, 269–282.

Digital Library

[23]

Patrick Cousot and Radhia Cousot. 1992. Comparing the Galois Connection and Widening/Narrowing Approaches to Abstract Interpretation. In PLILP (LNCS), Vol. 631. Springer, 269–295.

Digital Library

[24]

Patrick Cousot and Nicolas Halbwachs. 1978. Automatic Discovery of Linear Restraints Among Variables of a Program. In POPL. ACM, 84–96.

Digital Library

[25]

Pascal Cuoq, Benjamin Monate, Anne Pacalet, Virgile Prevosto, John Regehr, Boris Yakobowski, and Xuejun Yang. 2012. Testing Static Analyzers with Randomly Generated Programs. In NFM (LNCS), Vol. 7226. Springer, 120–125.

Digital Library

[26]

Catherine Dubois. 2000. Proving ML Type Soundness Within Coq. In TPHOLs (LNCS), Vol. 1869. Springer, 126–144.

Digital Library

[27]

Manuel Fähndrich and Francesco Logozzo. 2010. Static Contract Checking with Abstract Interpretation. In FoVeOOS (LNCS), Vol. 6528. Springer, 10–30.

[28]

Khalil Ghorbal, Eric Goubault, and Sylvie Putot. 2009. The Zonotope Abstract Domain Taylor1+. In CAV (LNCS), Vol. 5643. Springer, 627–633.

Digital Library

[29]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In PLDI. ACM, 213–223.

Digital Library

[30]

Eric Goubault and Sylvie Putot. 2006. Static Analysis of Numerical Algorithms. In SAS (LNCS), Vol. 4134. Springer, 18–34.

Digital Library

[31]

Julien Henry, David Monniaux, and Matthieu Moy. 2012. PAGAI: A Path Sensitive Static Analyser. Electr. Notes Theor. Comput. Sci. 289 (2012), 15–25.

Digital Library

[32]

Bertrand Jeannet and Antoine Miné. 2009. Apron: A Library of Numerical Abstract Domains for Static Analysis. In CAV (LNCS), Vol. 5643. Springer, 661– 667.

Digital Library

[33]

Jacques-Henri Jourdan, Vincent Laporte, Sandrine Blazy, Xavier Leroy, and David Pichardie. 2015. A Formally-Verified C Static Analyzer. In POPL. ACM, 247–259.

Digital Library

[34]

Timotej Kapus and Cristian Cadar. 2017. Automatic testing of symbolic execution engines via program generation and differential testing. In ASE. IEEE Computer Society, 590–600.

Digital Library

[35]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In PLDI. ACM, 216–226.

Digital Library

[36]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In OOPSLA. ACM, 386–399.

Digital Library

[37]

Xavier Leroy. 2009. Formal verification of a realistic compiler. CACM 52, 7 (2009), 107–115.

Digital Library

[38]

Benjamin Livshits, Manu Sridharan, Yannis Smaragdakis, Ondřej Lhoták, J. Nelson Amaral, Bor-Yuh Evan Chang, Samuel Z. Guyer, Uday P. Khedker, Anders Møller, and Dimitrios Vardoulakis. 2015. In Defense of Soundiness: A Manifesto. CACM 58 (2015), 44–46. Issue 2.

Digital Library

[39]

Francesco Logozzo and Manuel Fähndrich. 2010. Pentagons: A weakly relational abstract domain for the efficient validation of array accesses. Sci. Comput. Program. 75, 9 (2010), 796–807.

Digital Library

[40]

Magnus Madsen and Ondrej Lhoták. 2018. Safe and Sound Program Analysis with FLIX. In ISSTA. ACM. To appear.

Digital Library

[41]

Jan Midtgaard and Anders Møller. 2017. QuickChecking Static Analysis Properties. Softw. Test. Verif. Reliab. 27, 6 (2017).

[42]

Antoine Miné. 2006. The Octagon Abstract Domain. Higher Order Symbol. Comput. 19, 1 (2006), 31–100.

Digital Library

[43]

Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In ICSE. IEEE Computer Society, 75–84.

Digital Library

[44]

Thomas W. Reps, Shmuel Sagiv, and Greta Yorsh. 2004. Symbolic Implementation of the Best Transformer. In VMCAI (LNCS), Vol. 2937. Springer, 252–266.

[45]

Zhong Shao, Bratin Saha, Valery Trifonov, and Nikolaos Papaspyrou. 2002. A type system for certified binaries. In POPL. ACM, 217–232.

Digital Library

[46]

Gagandeep Singh, Markus Püschel, and Martin Vechev. 2015. Making Numerical Program Analysis Fast. In PLDI. ACM, 303–313.

Digital Library

[47]

Gagandeep Singh, Markus Püschel, and Martin Vechev. 2017. Fast Polyhedra Abstract Domain. In POPL. ACM, 46–59.

Digital Library

[48]

Gagandeep Singh, Markus Püschel, and Martin Vechev. 2018. A Practical Construction for Decomposing Numerical Abstract Domains. PACMPL 2, POPL (2018), 55:1–55:28.

Digital Library

[49]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding and analyzing compiler warning defects. In ICSE. ACM, 203–213.

Digital Library

[50]

H. Le Verge. 1992. A note on Chernikova’s Algorithm. Technical Report RR-1662. INRIA.

[51]

Shiyi Wei, Piotr Mardziel, Andrew Ruef, Jeffrey S. Foster, and Michael Hicks. 2018. Evaluating Design Tradeoffs in Numeric Static Analysis for Java. In ESOP (LNCS), Vol. 10801. Springer, 653–682.

Cited By

Monat ROuadjaout AMiné A(2025)Easing maintenance of academic static analyzersInternational Journal on Software Tools for Technology Transfer10.1007/s10009-024-00770-126:6(673-686)Online publication date: 14-Jan-2025
https://doi.org/10.1007/s10009-024-00770-1
He WDi PMing MZhang CSu TLi SSui Y(2024)Finding and Understanding Defects in Static Analyzers by Constructing Automated OraclesProceedings of the ACM on Software Engineering10.1145/36607811:FSE(1656-1678)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660781
Vishwanathan HShachnai MNarayana SNagarakatte SLee J(2022)Sound, precise, and fast abstract interpretation with tristate numbersProceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization10.1109/CGO53902.2022.9741267(254-265)Online publication date: 2-Apr-2022
https://dl.acm.org/doi/10.1109/CGO53902.2022.9741267
Show More Cited By

Index Terms

Automatically testing implementations of numerical abstract domains
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Pluggable abstract domains for analyzing embedded software
Proceedings of the 2006 LCTES Conference

Many abstract value domains such as intervals, bitwise, constants, and value-sets have been developed to support dataflow analysis. Different domains offer alternative tradeoffs between analysis speed and precision. Furthermore, some domains are a ...
Control-flow analysis of function calls and returns by abstract interpretation

Abstract interpretation techniques are used to derive a control-flow analysis for a simple higher-order functional language. The analysis approximates the interprocedural control-flow of both function calls and returns in the presence of first-class ...
Numerical static analysis with Soot
SOAP '13: Proceedings of the 2nd ACM SIGPLAN International Workshop on State Of the Art in Java Program analysis

Numerical static analysis computes an approximation of all the possible values that a numeric variable may assume, in any execution of the program. Many numerical static analyses have been proposed exploiting the theory of abstract interpretation, which ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

September 2018

955 pages

ISBN:9781450359375

DOI:10.1145/3238147

General Chair:
Marianne Huchard
University of Montpellier, France
,
Program Chairs:
Christian Kästner
Carnegie Mellon University, USA
,
Gordon Fraser
University of Passau, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
CNRS: Centre National De La Rechercue Scientifique
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE '18

Sponsor:

SIGAI
CNRS
SIGSOFT
IEEE-CS

ASE '18: 33rd ACM/IEEE International Conference on Automated Software Engineering

September 3 - 7, 2018

Montpellier, France

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
203
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Monat ROuadjaout AMiné A(2025)Easing maintenance of academic static analyzersInternational Journal on Software Tools for Technology Transfer10.1007/s10009-024-00770-126:6(673-686)Online publication date: 14-Jan-2025
https://doi.org/10.1007/s10009-024-00770-1
He WDi PMing MZhang CSu TLi SSui Y(2024)Finding and Understanding Defects in Static Analyzers by Constructing Automated OraclesProceedings of the ACM on Software Engineering10.1145/36607811:FSE(1656-1678)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660781
Vishwanathan HShachnai MNarayana SNagarakatte SLee J(2022)Sound, precise, and fast abstract interpretation with tristate numbersProceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization10.1109/CGO53902.2022.9741267(254-265)Online publication date: 2-Apr-2022
https://dl.acm.org/doi/10.1109/CGO53902.2022.9741267
Mansur MChristakis MWüstholz VSpinellis DGousios GChechik MDi Penta M(2021)Metamorphic testing of Datalog enginesProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468573(639-650)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468573
Winterer DZhang CSu ZDonaldson ATorlak E(2020)Validating SMT solvers via semantic fusionProceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation10.1145/3385412.3385985(718-730)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3385412.3385985
Taneja JLiu ZRegehr JMars JTang LXue JWu P(2020)Testing static analyses for precision and soundnessProceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization10.1145/3368826.3377927(81-93)Online publication date: 22-Feb-2020
https://dl.acm.org/doi/10.1145/3368826.3377927
Casso IMorales JLópez-García PHermenegildo M(2020)Testing Your (Static Analysis) TruthsLogic-Based Program Synthesis and Transformation10.1007/978-3-030-68446-4_14(271-292)Online publication date: 7-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-68446-4_14
Zhang CSu TYan YZhang FPu GSu ZDumas MPfahl DApel SRusso A(2019)Finding and understanding bugs in software model checkersProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338932(763-773)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338932
Tolksdorf SLehmann DPradel MZhang DMøller A(2019)Interactive metamorphic testing of debuggersProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3293882.3330567(273-283)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.1145/3293882.3330567
Klinger CChristakis MWüstholz VZhang DMøller A(2019)Differentially testing soundness and precision of program analyzersProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3293882.3330553(239-250)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.1145/3293882.3330553

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten