Article

Pex: white box test generation for .NET

Authors:

Nikolai Tillmann,

Jonathan De HalleuxAuthors Info & Claims

TAP'08: Proceedings of the 2nd international conference on Tests and proofs

Pages 134 - 153

Published: 09 April 2008 Publication History

Abstract

Pex automatically produces a small test suite with high code coverage for a .NET program. To this end, Pex performs a systematic program analysis (using dynamic symbolic execution, similar to path-bounded model-checking) to determine test inputs for Parameterized Unit Tests. Pex learns the program behavior by monitoring execution traces. Pex uses a constraint solver to produce new test inputs which exercise different program behavior. The result is an automatically generated small test suite which often achieves high code coverage. In one case study, we applied Pex to a core component of the .NET runtime which had already been extensively tested over several years. Pex found errors, including a serious issue.

References

[1]

Anand, S., Godefroid, P., Tillmann, N.: Demand-driven compositional symbolic execution. Technical Report MSR-TR-2007-138, Microsoft Research, Redmond, WA (October 2007).

[2]

Anand, S., Pasareanu, C.S., Visser, W.: Jpf-se: A symbolic execution extension to java pathfinder. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 134-138. Springer, Heidelberg (2007).

Digital Library

[3]

Bjorner, N., de Moura, L.: Z3: An efficient SMT solver (2007), http://research.microsoft.com/projects/Z3

[4]

Boshernitsan, M., Doong, R., Savoia, A.: From daikon to agitator: lessons and challenges in building a commercial tool for developer testing. In: ISSTA 2006: Proceedings of the 2006 international symposium on Software testing and analysis, pp. 169-180. ACM Press, New York (2006).

Digital Library

[5]

Brace, K.S., Rudell, R.L., Bryant, R.E.: Efficient implementation of a BDD package. In: DAC 1990: Proceedings of the 27th ACM/IEEE conference on Design automation, pp. 40-45. ACM Press, New York (1990).

Digital Library

[6]

Cadar, C., Ganesh, V., Pawlowski, P.M., Dill, D.L., Engler, D.R.: Exe: automatically generating inputs of death. In: CCS 2006: Proceedings of the 13th ACM conference on Computer and communications security, pp. 322-335. ACM Press, New York (2006).

Digital Library

[7]

Csallner, C., Tillmann, N., Smaragdakis, Y.: Dysy: Dynamic symbolic execution for invariant inference. Technical Report MSR-TR-2007-151, Microsoft Research, Redmond, WA (November 2007).

[8]

Engler, D., Dunbar, D.: Under-constrained execution: making automatic code destruction easy and scalable. In: ISSTA 2007: Proceedings of the 2007 international symposium on Software testing and analysis, pp. 1-4. ACM, New York (2007).

Digital Library

[9]

Ernst, M.D., Perkins, J.H., Guo, P.J., McCamant, S., Pacheco, C., Tschantz, M.S., Xiao, C.: The Daikon system for dynamic detection of likely invariants. Science of Computer Programming (2007).

Digital Library

[10]

Flanagan, C., Leino, K.R.M., Lillibridge, M., Nelson, G., Saxe, J.B., Stata, R.: Extended static checking for Java. In: Proc. the ACM SIGPLAN 2002 Conference on Programming language design and implementation, pp. 234-245. ACM Press, New York (2002).

Digital Library

[11]

Godefroid, P.: Compositional dynamic test generation. In: POPL 2007: Proceedings of the 34th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pp. 47-54. ACM Press, New York (2007).

Digital Library

[12]

Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. SIGPLAN Notices 40(6), 213-223 (2005).

Digital Library

[13]

Godefroid, P., Levin, M.Y., Molnar, D.: Automated whitebox fuzz testing. Technical Report MSR-TR-2007-58, Microsoft Research, Redmond, WA (May 2007).

[14]

Grieskamp, W., Tillmann, N., Schulte, W.: XRT - Exploring Runtime for .NET - Architecture and Applications. In: SoftMC 2005: Workshop on Software Model Checking, July 2005. Electronic Notes in Theoretical Computer Science (2005).

Digital Library

[15]

E. International. Standard ECMA-335, Common Language Infrastructure (CLI) (June 2006).

[16]

King, J.C.: Symbolic execution and program testing. Commun. ACM 19(7), 385- 394 (1976).

Digital Library

[17]

Korel, B.: A dynamic approach of test data generation. In: IEEE Conference On Software Maintenance, November 1990, pp. 311-317 (1990).

[18]

Korel, B., Al-Yami, A.M.: Assertion-oriented automated test data generation. In: Proc. the 18th international conference on Software engineering, pp. 71-80. IEEE Computer Society, Los Alamitos (1996).

Digital Library

[19]

Majumdar, R., Sen, K.: Latest: Lazy dynamic test input generation. Technical Report UCB/EECS-2007-36, EECS Department, University of California, Berkeley (Mar 2007).

[20]

Two, M.C., Poole, C., Cansdale, J., Feldman, G., Newkirk, J.W., Vorontsov, A.A., Craig, P.A.: NUnit, http://www.nunit.org/

[21]

Microsoft. Net framework general reference - profiling (unmanaged api reference), http://msdn2.microsoft.com/en-us/library/ms404386.aspx

[22]

Microsoft. Visual Studio Team System, Team Edition for Testers, http://msdn2.microsoft.com/en-us/vsts2008/products/bb933754.aspx

[23]

Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T.: Feedback-directed random test generation. In: ICSE 2007, Proceedings of the 29th International Conference on Software Engineering, Minneapolis, MN, USA, May 23-25 (2007).

Digital Library

[24]

Pex development team. Pex (2007), http://research.microsoft.com/Pex

[25]

Saff, D., Boshernitsan, M., Ernst, M.D.: Theories in practice: Easy-to-write specifications that catch bugs. Technical Report MIT-CSAIL-TR-2008-002, MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, January 14 (2008).

[26]

Sen, K., Agha, G.: CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools. In: Ball, T., Jones, R.B. (eds.) CAV 2006. LNCS, vol. 4144, pp. 419-423. Springer, Heidelberg (2006).

Digital Library

[27]

Sen, K., Marinov, D., Agha, G.: Cute: a concolic unit testing engine for c. In: ESEC/FSE-13: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, pp. 263-272. ACM Press, New York (2005).

Digital Library

[28]

Tillmann, N., Schulte, W.: Parameterized unit tests. In: Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 253-262. ACM, New York (2005).

Digital Library

[29]

Tillmann, N., Schulte, W.: Unit tests reloaded: Parameterized unit testing with symbolic execution. IEEE Software 23(4), 38-47 (2006).

Digital Library

Cited By

Kluban MMannan MYoussef A(2024)On Detecting and Measuring Exploitable JavaScript Functions in Real-world ApplicationsACM Transactions on Privacy and Security10.1145/363025327:1(1-37)Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3630253
Smaragdakis YGrech NLagouvardos STriantafyllou KTsatiris I(2021)Symbolic value-flow static analysis: deep, precise, complete modeling of Ethereum smart contractsProceedings of the ACM on Programming Languages10.1145/34855405:OOPSLA(1-30)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3485540
Wang SShrestha NSubburaman AWang JWei MNagappan N(2021)Automatic Unit Test Generation for Machine Learning LibrariesProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00138(1548-1560)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00138
Show More Cited By

Index Terms

Pex: white box test generation for .NET
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Index terms have been assigned to the content through auto-classification.

Recommendations

Pex--An Intelligent Assistant for Rigorous Developer Testing
ICECCS '07: Proceedings of the 12th IEEE International Conference on Engineering Complex Computer Systems

Pex takes test-driven development to the next level. Pex analyzes .NET applications. From a parameterized unit test, which serves as a specification, it automatically produces traditional unit tests cases with high code coverage. Moreover, when a ...
Parameterized unit testing with Pex
TAP'08: Proceedings of the 2nd international conference on Tests and proofs

This hands-on tutorial will teach the principles of Parameterized Unit Testing [5,4] with Pex [2], an automatic test input generator for .NET which performs a systematic program analysis, similar to path bounded model-checking.

A parameterized unit test ...
White-box testing of behavioral web service contracts with Pex
TAV-WEB '08: Proceedings of the 2008 workshop on Testing, analysis, and verification of web services and applications

A web service exposes a public API that can be accessed by potentially hostile clients over the internet. Pex, a white-box test generation tool for .NET, can automatically create test inputs that cover comer cases of a web service implemented in .NET, ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

TAP'08: Proceedings of the 2nd international conference on Tests and proofs

April 2008

192 pages

ISBN:354079123X

Editors:
Bernhard Beckert
University of Koblenz-Landau, Dept. of Computer Science, Koblenz, Germany
,
Reiner Hähnle
Chalmers University of Technology, Dept. of Computer Science and Engineering, Göteborg, Sweden

Sponsors

Microsoft Research (USA)
ETH Zurich
University of Koblenz-Landau
Chalmers University of Technology

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 09 April 2008

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

296
Total Citations
View Citations
24
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kluban MMannan MYoussef A(2024)On Detecting and Measuring Exploitable JavaScript Functions in Real-world ApplicationsACM Transactions on Privacy and Security10.1145/363025327:1(1-37)Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3630253
Smaragdakis YGrech NLagouvardos STriantafyllou KTsatiris I(2021)Symbolic value-flow static analysis: deep, precise, complete modeling of Ethereum smart contractsProceedings of the ACM on Programming Languages10.1145/34855405:OOPSLA(1-30)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3485540
Wang SShrestha NSubburaman AWang JWei MNagappan N(2021)Automatic Unit Test Generation for Machine Learning LibrariesProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00138(1548-1560)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00138
Molina FPonzio PAguirre NFrias M(2021)EvoSpexProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00112(1223-1235)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00112
Melo LRibeiro RGuimarães BPereira F(2020)Type Inference for CACM Transactions on Programming Languages and Systems10.1145/342147242:3(1-71)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3421472
Wu ZJohnson EYang WBastani OSong DPeng JXie TDumas MPfahl DApel SRusso A(2019)REINAM: reinforcement learning for input-grammar inferenceProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338958(488-498)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338958
Kim YHong SKim MDumas MPfahl DApel SRusso A(2019)Target-driven compositional concolic testing with function summary refinement for effective bug detectionProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338934(16-26)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338934
Liew DCadar CDonaldson AStinnett JDumas MPfahl DApel SRusso A(2019)Just fuzz it: solving floating-point constraints using coverage-guided fuzzingProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338921(521-532)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338921
Ahmadi RDingel JDumas MPfahl DApel SRusso A(2019)Concolic testing for models of state-based systemsProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338908(4-15)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338908
Schuster CFlanagan C(2019)IDVECompanion Proceedings of the 3rd International Conference on the Art, Science, and Engineering of Programming10.1145/3328433.3328453(1-16)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1145/3328433.3328453
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents