research-article

Generating Test Cases for Programs that Are Coded against Interfaces and Annotations

Authors:

Christoph CsallnerAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 23, Issue 3

Article No.: 21, Pages 1 - 38

https://doi.org/10.1145/2544135

Published: 02 June 2014 Publication History

Abstract

Automatic test case generation for software programs is very powerful but suffers from a key limitation. That is, most current test case generation techniques fail to cover testee code when covering that code requires additional pieces of code not yet part of the program under test. To address some of these cases, the Pex state-of-the-art test case generator can generate basic mock code. However, current test case generators cannot handle cases in which the code under test uses multiple interfaces, annotations, or reflection.

To cover such code in an object-oriented setting, we describe a novel technique for generating test cases and mock classes. The technique consists of collecting constraints on interfaces, annotations, and reflection, combining them with program constraints collected during dynamic symbolic execution, encoding them in a constraint system, solving them with an off-the-shelf constraint solver, and mapping constraint solutions to test cases and custom mock classes. We demonstrate the value of this technique on open-source applications. Our approach covered such third-party code with generated mock classes, while competing approaches failed to cover the code and sometimes produced unintended side-effects such as filling the screen with dialog boxes and writing into the file system.

References

[1]

Saswat Anand, Alessandro Orso, and Mary Jean Harrold. 2007. Type-dependence analysis and program transformation for symbolic execution. In Proceeding of the 13^th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'07). Springer, 117--133.

Digital Library

[2]

Mike Barnett, K. Rustan M. Leino, and Wolfram Schulte. 2004. The Spec&num; programming system: An overview. In Proceedings of the International Workshop on the Construction and Analysis of Safe, Secure, and Interoperable Smart Devices (CASSIS'04). Springer, 49--69.

Digital Library

[3]

Kent Beck and Erich Gamma. 1998. Test infected: Programmers love writing tests. Java Rep. 3, 7, 37--50.

[4]

Boris Beizer. 1990. Software Testing Techniques 2^nd Ed. Van Nostrand Reinhold.

Digital Library

[5]

Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010. A few billion lines of code later: Using static analysis to find bugs in the real world. Comm. ACM 53, 2, 66--75.

Digital Library

[6]

Cédric Beust and Hani Suleiman. 2007. Next Generation Java Testing: TestNG and Advanced Concepts. Addison-Wesley.

Digital Library

[7]

Eric Bodden, Andreas Sewe, Jan Sinschek, and Mira Mezini. 2010. Taming reflection (extended version). Tech. rep. TUD-CS-2010-0066. TU Darmstadt. http://www.bodden.de/pubs/tr-tamiflex.pdf.

[8]

Eric Bodden, Andreas Sewe, Jan Sinschek, and Mira Mezini. 2011. Taming reflection: Aiding static analysis in the presence of reflection and custom class loaders. In Proceedings of the 33^rd ACM/IEEE International Conference on Software Engineering (ICSE'11). ACM Press, New York, 241--250.

Digital Library

[9]

Grady Booch, James Rumbaugh, and Ivar Jacobson. 2005. The Unified Modeling Language User Guide 2^nd Ed. Addison-Wesley.

Digital Library

[10]

Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated testing based on java predicates. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'02). ACM Press, 123--133.

Digital Library

[11]

Jörg Brauer and Axel Simon. 2012. Inferring definite counterexamples through under-approximation. In Proceedings of the 4^th International NASA Formal Methods Symposium (NFM'12). Springer, 54--69.

Digital Library

[12]

Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8^th USENIX Symposium on Operating Systems Design and Implementation (OSDI'08). USENIX, 209--224.

Digital Library

[13]

Cristian Cadar, Vijay Ganesh, Peter M. Pawlowski, David L. Dill, and Dawson R. Engler. 2006. EXE: Automatically generating inputs of death. In Proceedings of the 13^th ACM Conference on Computer and Communications Security (CCS'06). ACM Press, New York, 322--335.

Digital Library

[14]

Satish Chandra, Stephen J. Fink, and Manu Sridharan. 2009. Snugglebug: A powerful approach to weakest preconditions. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'09). ACM Press, New York, 363--374.

Digital Library

[15]

Lori A. Clarke. 1976. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Engin. 2, 3, 215--222.

Digital Library

[16]

David R. Cok and Joseph R. Kiniry. 2004. ESC/Java2: Uniting esc/java and jml: Progress and issues in building and using esc/java2. Tech. rep. NIII-R0413, Nijmegen Institute for Computing and Information Science.

[17]

Christian Collberg, Ginger Myles, and Michael Stepp. 2007. An empirical study of Java bytecode programs. Softw. Pract. Exper. 37, 6, 581--641.

Digital Library

[18]

Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: An automatic robustness tester for Java. Softw. Pract. Exper. 34, 11, 1025--1050.

Digital Library

[19]

Christoph Csallner and Yannis Smaragdakis. 2005. Check 'n' Crash: Combining static checking and testing. In Proceedings of the 27^th ACM/IEEE International Conference on Software Engineering (ICSE'05). ACM Press, New York, 422--431.

Digital Library

[20]

Christoph Csallner, Yannis Smaragdakis, and Tao Xie. 2008. DSD-Crasher: A hybrid analysis tool for bug finding. ACM Trans. Softw. Engin. Methodol. 17, 2, 1--37.

Digital Library

[21]

Jonathan de Halleux and Nikolai Tillmann. 2010. Moles: Tool-assisted environment isolation with closures. In Proceedings of the 48^th International Conference on Objects, Models, Components, Patterns (TOOLS'10). Springer, 253--270.

Digital Library

[22]

Leonardo de Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. In Proceedings of the 14^th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'08). Springer, 337--340.

Digital Library

[23]

Hyunsook Do, Sebastian G. Elbaum, and Gregg Rothermel. 2005. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empir. Softw. Engin. 10, 4, 405--435.

Digital Library

[24]

Bassem Elkarablieh, Patrice Godefroid, and Michael Y. Levin. 2009. Precise pointer reasoning for dynamic test generation. In Proceedings of the 18^th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'09). ACM Press, New York, 129--140.

Digital Library

[25]

Cormac Flanagan, K. Rustan M. Leino, Mark Lillibridge, Greg Nelson, James B. Saxe, and Raymie Stata. 2002. Extended static checking for Java. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'02). ACM Press, New York, 234--245.

Digital Library

[26]

Steve Freeman, Tim Mackinnon, Nat Pryce, and Joe Walnes. 2004a. jMock: Supporting responsibility-based design with mock objects. In Companion to the 19^th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA'04). ACM Press, New York, 4--5.

Digital Library

[27]

Steve Freeman, Tim Mackinnon, Nat Pryce, and Joe Walnes. 2004b. Mock roles, not objects. In Companion to the 19^th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA'04). ACM Press, New York, 236--246.

Digital Library

[28]

Steve Freeman and Nat Pryce. 2006. Evolving an embedded domain-specific language in java. In Companion to the 21^st ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA'06). ACM Press, New York, 855--865.

Digital Library

[29]

Stefan J. Galler, Martin Weiglhofer, and Franz Wotawa. 2010. Synthesize it: From design by contract to meaningful test input data. In Proceedings of the 8^th IEEE International Conference on Software Engineering and Formal Methods (SEFM'10). IEEE, 286--295.

Digital Library

[30]

Patrice Godefroid. 2007. Compositional dynamic test generation. In Proceedings of the 34^th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL'07). ACM Press, New York, 47--54.

Digital Library

[31]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'05). ACM Press, New York, 213--223.

Digital Library

[32]

Patrice Godefroid, Michael Y. Levin, and David Molnar. 2008. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium (NDSS'08). The Internet Society.

[33]

James Gosling, Bill Joy, Guy L. Steele, and Gilad Bracha. 2005. The Java Language Specification 3^rd Ed. Prentice Hall.

[34]

Mark Hapner, Rich Burridge, Rahul Sharma, and Joseph Fialli. 2002. Java Message Service: Version 1.1. Sun Microsystems.

[35]

Michael Hind. 2001. Pointer analysis: Haven't we solved this problem yet&quest; In Proceedings of the ACM SIGPLANSIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE'01). ACM Press, New York, 54--61.

Digital Library

[36]

Mainul Islam and Christoph Csallner. 2010. Dsc+Mock: A test case + mock class generator in support of coding against interfaces. In Proceedings of the 8^th International Workshop on Dynamic Analysis (WODA'10). ACM Press, New York, 26--31.

Digital Library

[37]

Hojun Jaygarl, Sunghun Kim, Tao Xie, and Carl K. Chang. 2010. OCAT: Object capture-based automated testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'10). ACM Press, New York, 159--170.

Digital Library

[38]

Gregor Kiczales, John Lamping, Anurag Mendhekar, Chris Maeda, Cristina Videira Lopes, Jean-Marc Loingtier, and John Irwin. 1997. Aspect-oriented programming. In Proceedings of the 11^th European Conference on Object-Oriented Programming (ECOOP'97). 220--242.

[39]

James C. King. 1976. Symbolic execution and program testing. Comm. ACM 19, 7, 385--394.

Digital Library

[40]

Joseph R. Kiniry. 2004. The logics and calculi of ESC/Java2. http://kindsoftware.com/products/opensource/ESCJava2/ESCTools/docs/Escjava2-Logics.pdf.

[41]

Mika Mantyla, Juha Itkonen, and Joonas Iivonen. 2012. Who tested my software&quest; Testing as an organizationally cross-cutting activity. Softw. Quality J. 20, 1, 145--172.

Digital Library

[42]

Madhuri R. Marri, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. 2009. An empirical study of testing file-system-dependent software with mock objects. In Proceedings of the 4^th International Workshop on Automation of Software Test (AST'09). IEEE, 149--153.

[43]

Steve Mcconnell. 2004. Code Complete 2^nd Ed. Microsoft Press.

[44]

Gerard Meszaros. 2007. xUnit Test Patterns: Refactoring Test Code. Addison-Wesley.

Digital Library

[45]

Audris Mockus, Roy T. Fielding, and James D. Herbsleb. 2002. Two case studies of open source software development: Apache and Mozilla. ACM Trans. Softw. Engin. Methodol. 11, 3, 309--346.

Digital Library

[46]

National Institute of Standards and Technology. 2002. The economic impacts of inadequate infrastructure for software testing: Planning report 02-3. http://www.nist.gov/director/planning/upload/report02- 3.pdf.

[47]

Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29^th ACM/IEEE International Conference on Software Engineering (ICSE'07). 75--84.

Digital Library

[48]

Benny Pasternak, Shmuel S. Tyszberowicz, and Amiram Yehudai. 2009. GenUTest: A unit test and mock aspect generation tool. Int. J. Softw. Tools Technol. Transfer 11, 4, 273--290.

Digital Library

[49]

Mauro Pezze and Michal Young. 2007. Software Testing and Analysis: Process, Principles and Techniques. Wiley.

[50]

David Saff, Shay Artzi, Jeff H. Perkins, and Michael D. Ernst. 2005. Automatic test factoring for Java. In Proceedings of the 20^th IEEE/ACM International Conference on Automated Software Engineering (ASE'05). ACM Press, New York, 114--123.

Digital Library

[51]

David Saff and Michael D. Ernst. 2004. Mock object creation for test factoring. In Proceedings of the ACM SIGPLANSIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE'04). ACM Press, New York, 49--51.

Digital Library

[52]

Suresh Thummalapenta, Jonathan de Halleux, Nikolai Tillmann, and Scott Wadsworth. 2010. DyGen: Automatic generation of high-coverage tests via mining gigabytes of dynamic traces. In Proceedings of the 4^th International Conference on Tests and Proofs (TAP'10). Springer, 77--93.

Digital Library

[53]

Nikolai Tillmann and Jonathan de Halleux. 2008. Pex - White box test generation for .net. In Proceedings of the 2^nd International Conference on Tests And Proofs (TAP'08). Springer, 134--153.

Digital Library

[54]

Nikolai Tillmann and Wolfram Schulte. 2006a. Mock-object generation with behavior. In Proceedings of the 21^st IEEE/ACM International Conference on Automated Software Engineering (ASE'06). 365--368.

Digital Library

[55]

Nikolai Tillmann and Wolfram Schulte. 2006b. Unit tests reloaded: Parameterized unit testing with symbolic execution. IEEE Softw. 23, 4, 38--47.

Digital Library

[56]

Gina D. Venolia, Robert Deline, and Thomas Latoza. 2005. Software development at microsoft observed. Tech. rep. MSR-TR-2005-140, Microsoft Research.

[57]

Stefan Wagner, Jan Jurjens, Claudia Koller, and Peter Trischberger. 2005. Comparing bug finding tools with reviews and tests. In Proceedings of the 17^th IFIP TC6/WG 6.1 International Conference on Testing of Communicating Systems (TestCom'05). Springer, 40--55.

Digital Library

[58]

Laurie Williams, Gabe Brown, Adam Meltzer, and Nachiappan Nagappan. 2011. Scrum + engineering practices: Experiences of three microsoft teams. In Proceedings of the 5^th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM'11). IEEE, 463--471.

Digital Library

[59]

Laurie Williams, Gunnar Kudrjavets, and Nachiappan Nagappan. 2009. On the effectiveness of unit test automation at microsoft. In Proceedings of the 20^th IEEE International Symposium on Software Reliability Engineering (ISSRE'09). IEEE, 81--89.

Digital Library

[60]

Xusheng Xiao, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. 2011. Precise identification of problems for structural test generation. In Proceedings of the 33^rd ACM/IEEE International Conference on Software Engineering (ICSE'11). ACM Press, New York, 611--620.

Digital Library

[61]

Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. 2009. Fitness-guided path exploration in dynamic symbolic execution. In Proceedings of the International Conference on Dependable Systems and Networks (DSN'09). IEEE, 359--368.

[62]

Andy Zaidman, Bart van Rompaey, Arie van Deursen, and Serge Demeyer. 2011. Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empir. Softw. Engin. 16, 3, 325--364.

Digital Library

[63]

Misha Zitser, Richard Lippmann, and Tim Leek. 2004. Testing static analysis tools using exploitable buffer overflows from open source code. In Proceedings of the 12^th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE'04). ACM Press, New York, 97--106.

Digital Library

Cited By

Lahiri SPotanin A(2022)Verification of Programs with Concealed ComponentsCompanion Proceedings of the 2022 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity10.1145/3563768.3565551(43-46)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563768.3565551
Fazzini MChoi CCopia JLee GKakehi YGorla AOrso ADwyer MDamian DZeller A(2022)Use of test doubles in Android testingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510175(2266-2278)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510175
Mayeda MAndrews A(2021)Evaluating software testing techniques: A systematic mapping study10.1016/bs.adcom.2021.01.002(41-114)Online publication date: 2021
https://doi.org/10.1016/bs.adcom.2021.01.002
Show More Cited By

Index Terms

Generating Test Cases for Programs that Are Coded against Interfaces and Annotations

Recommendations

Dsc+Mock: a test case + mock class generator in support of coding against interfaces
WODA '10: Proceedings of the Eighth International Workshop on Dynamic Analysis

Coding against interfaces is a powerful technique in object-oriented programming. It decouples code and enables independent development. However, code decoupled via interfaces poses additional challenges for testing and dynamic execution, as not all ...
Achieving scalable mutation-based generation of whole test suites

Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...
Generating User Acceptance Test Plans from Test Cases
COMPSAC '07: Proceedings of the 31st Annual International Computer Software and Applications Conference - Volume 02

There are numerous studies on generating test cases from classifications. These test cases consists of related classes from different classifications. On the other hand, a User Acceptance Test Plan (UAT) consist of test plans. Each of these test plans ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 23, Issue 3

May 2014

230 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2628068

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 June 2014

Accepted: 01 November 2013

Revised: 01 November 2013

Received: 01 October 2012

Published in TOSEM Volume 23, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
596
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lahiri SPotanin A(2022)Verification of Programs with Concealed ComponentsCompanion Proceedings of the 2022 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity10.1145/3563768.3565551(43-46)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563768.3565551
Fazzini MChoi CCopia JLee GKakehi YGorla AOrso ADwyer MDamian DZeller A(2022)Use of test doubles in Android testingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510175(2266-2278)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510175
Mayeda MAndrews A(2021)Evaluating software testing techniques: A systematic mapping study10.1016/bs.adcom.2021.01.002(41-114)Online publication date: 2021
https://doi.org/10.1016/bs.adcom.2021.01.002
Shrestha SPanda SCsallner CTichy WMinku L(2018)Complementing machine learning classifiers via dynamic symbolic executionProceedings of the 6th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering10.1145/3194104.3194111(15-20)Online publication date: 28-May-2018
https://dl.acm.org/doi/10.1145/3194104.3194111
Arcuri AFraser GJust R(2017)Private API Access and Functional Mocking in Automated Unit Test Generation2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2017.19(126-137)Online publication date: Mar-2017
https://doi.org/10.1109/ICST.2017.19
Landman DSerebrenik AVinju JUchitel SOrso ARobillard M(2017)Challenges for static analysis of Java reflectionProceedings of the 39th International Conference on Software Engineering10.1109/ICSE.2017.53(507-518)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICSE.2017.53
Córdoba-Sánchez Ide Lara J(2016)AnnComputer Languages, Systems and Structures10.1016/j.cl.2016.02.00245:C(164-190)Online publication date: 1-Apr-2016
https://dl.acm.org/doi/10.1016/j.cl.2016.02.002
Sherman EGarvin BDwyer M(2015)Deciding Type-Based Partial-Order Constraints for Path-Sensitive AnalysisACM Transactions on Software Engineering and Methodology10.1145/275597124:3(1-33)Online publication date: 13-May-2015
https://dl.acm.org/doi/10.1145/2755971
Li KReichenbach CCsallner CSmaragdakis Y(2014)Residual InvestigationACM Transactions on Software Engineering and Methodology10.1145/265620124:2(1-32)Online publication date: 23-Dec-2014
https://dl.acm.org/doi/10.1145/2656201

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents