June: A Type Testability Transformation for Improved ATG Performance
Pages 274 - 284
Abstract
Strings are universal containers: they are flexible to use, abundant in code, and difficult to test. String-controlled programs are programs that make branching decisions based on string input. Automatically generating valid test inputs for these programs considering only character sequences rather than any underlying string-encoded structures, can be prohibitively expensive. We present June, a tool that enables Java developers to expose any present latent string structure to test generation tools. June is an annotation-driven testability transformation and an extensible library, JuneLib, of structured string definitions. The core JuneLib definitions are empirically derived and provide templates for all structured strings in our test set. June takes lightly annotated source code and injects code that permits an automated test generator (ATG) to focus on the creation of mutable substrings inside a structured string. Using June costs the developer little, with an average of 2.1 annotations per string-controlled class. June uses standard Java build tools and therefore deploys seamlessly within a Java project. By feeding string structure information to an ATG tool, June dramatically reduces wasted effort; branches are effortlessly covered that would otherwise be extremely difficult, or impossible, to cover. This waste reduction both increases and speeds coverage. EvoSuite, for example, achieves the same coverage on June-ed classes in 1 minute, on average, as it does in 9 minutes on the un-June-ed class. These gains increase over time. On our corpus, June-ing a program compresses 24 hours of execution time into ca. 2 hours. We show that many ATG tools can reuse the same June-ed code: a few June annotations, a one-off cost, benefit many different testing regimes.
References
[1]
[n. d.]. java-afl. https://github.com/Barro/java-afl. Accessed: 2022-11-11
[2]
Accessed: 2022-11-11. Peach Fuzzer Platform. http://www.peachfuzzer.com/products/peach-platform/.
[3]
Mohammad Alshraideh and Leonardo Bottaci. 2006. Search-based software test data generation for string data using program-specific search operators. Softw. Test., Verif. Reliab., 16 (2006), 09, 175–203. https://doi.org/10.1002/stvr.354
[4]
Paul Ammann and Jeff Offutt. 2016. Introduction to Software Testing (2nd ed.). Cambridge University Press, New York, NY, USA. isbn:1107172012, 9781107172012
[5]
Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd international conference on software engineering. 1–10.
[6]
Vaggelis Atlidakis, Roxana Geambasu, Patrice Godefroid, Marina Polishchuk, and Baishakhi Ray. 2020. Pythia: Grammar-Based Fuzzing of REST APIs with Coverage-guided Feedback and Learning-based Mutations. 05.
[7]
Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2017. Synthesizing Program Input Grammars. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2017). ACM, New York, NY, USA. 95–110. isbn:978-1-4503-4988-8 https://doi.org/10.1145/3062341.3062349
[8]
Michael Beyene and James H. Andrews. 2012. Generating String Test Data for Code Coverage. Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012, 270–279. isbn:978-1-4577-1906-6 https://doi.org/10.1109/ICST.2012.107
[9]
Tim Blazytko, Matt Bishop, Cornelius Aschermann, Justin Cappos, Moritz Schlögel, Nadia Korshun, Ali Abbasi, Marco Schweighauser, Sebastian Schinzel, and Sergej Schumilo. 2019. $GRIMOIRE$: Synthesizing structure while fuzzing. In 28th USENIX Security Symposium (USENIX Security 19). 1985–2002.
[10]
Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed Greybox Fuzzing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS ’17). ACM, New York, NY, USA. 2329–2344. isbn:978-1-4503-4946-8 https://doi.org/10.1145/3133956.3134020
[11]
Dan Bruce, Héctor D Menéndez, Earl T Barr, and David Clark. 2020. Ant colony optimization for object-oriented unit test generation. In International Conference on Swarm Intelligence. 29–41.
[12]
José Campos, Annibale Panichella, and Gordon Fraser. 2019. EvoSuite at the SBST 2019 Tool Competition. In Proceedings of the 12th International Workshop on Search-Based Software Testing (SBST ’19).
[13]
Carl Chapman and Kathryn T. Stolee. 2016. Exploring Regular Expression Usage and Context in Python. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016). Association for Computing Machinery, New York, NY, USA. 282–293. isbn:9781450343909 https://doi.org/10.1145/2931037.2931073
[14]
David Coppit and Jiexin Lian. 2005. Yagg: An easy-to-use generator for structured test inputs. 20th IEEE/ACM International Conference on Automated Software Engineering, ASE 2005, 356–359. https://doi.org/10.1145/1101908.1101969
[15]
Santanu Kumar Dash, Miltiadis Allamanis, and Earl T Barr. 2018. RefiNym: using names to refine types. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 107–117.
[16]
James C. Davis, Christy A. Coghlan, Francisco Servant, and Dongyoon Lee. 2018. The Impact of Regular Expression Denial of Service (ReDoS) in Practice: An Empirical Study at the Ecosystem Scale. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA. 246–256. isbn:9781450355735 https://doi.org/10.1145/3236024.3236027
[17]
Ivan Enderlin, Frédéric Dadeau, Alain Giorgetti, and Fabrice Bouquet. 2012. Grammar-Based Testing using Realistic Domains in PHP. Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012, 04, https://doi.org/10.1109/ICST.2012.136
[18]
Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419.
[19]
Gordon Fraser and Andrea Arcuri. 2013. Whole test suite generation. IEEE Transactions on Software Engineering, 39, 2 (2013), 276–291.
[20]
Gordon Fraser and Andrea Arcuri. 2014. A Large Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Transactions on Software Engineering and Methodology (TOSEM), 24, 2 (2014), 8.
[21]
Gordon Fraser, José Miguel Rojas, and Andrea Arcuri. 2018. Evosuite at the SBST 2018 Tool Competition. In Proceedings of the 11th International Workshop on Search-Based Software Testing (SBST ’18). Association for Computing Machinery, New York, NY, USA. 34–37. isbn:9781450357418 https://doi.org/10.1145/3194718.3194729
[22]
Patrice Godefroid, Adam Kiezun, and Michael Y. Levin. 2007. Grammar-based Whitebox Fuzzing. 10. https://www.microsoft.com/en-us/research/publication/ grammar-based-whitebox-fuzzing/
[23]
Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In In Programming Language Design and Implementation (PLDI.
[24]
Rahul Gopinath, Björn Mathis, Mathias Höschele, Alexander Kampmann, and Andreas Zeller. 2018. Sample-Free Learning of Input Grammars for Comprehensive Software Fuzzing. arXiv preprint arXiv:1810.08289.
[25]
Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, André Baresel, and Marc Roper. 2004. Testability transformation. IEEE Transactions on Software Engineering, 30, 1 (2004), 3–16.
[26]
Matthias Höschele and Andreas Zeller. 2016. Mining Input Grammars from Dynamic Taints. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). ACM, New York, NY, USA. 720–725. isbn:978-1-4503-3845-5 https://doi.org/10.1145/2970276.2970321
[27]
Matthias Höschele and Andreas Zeller. 2017. Mining input grammars with autogram. In Proceedings of the 39th International Conference on Software Engineering Companion. 31–34.
[28]
David Kelly, Mark Marron, David Clark, and Earl T. Barr. 2019. SafeStrings: Representing Strings as Structured Data. CoRR, abs/1904.11254 (2019), arxiv:1904.11254
[29]
B. Korel. 1990. Automated Software Test Data Generation. IEEE Trans. Softw. Eng., 16, 8 (1990), Aug., 870–879. issn:0098-5589 https://doi.org/10.1109/32.57624
[30]
Leonidas Lampropoulos, Michael Hicks, and Benjamin C Pierce. 2019. Coverage guided, property based testing. Proceedings of the ACM on Programming Languages, 3, OOPSLA (2019), 1–29.
[31]
EvoSuite Maintainers. 2022. EvoSuite: Automatic Test Suite Generation for Java. https://www.evosuite.org/
[32]
Björn Mathis, Rahul Gopinath, Michaël Mera, Alexander Kampmann, Matthias Höschele, and Andreas Zeller. 2019. Parser-directed fuzzing. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. 548–560.
[33]
Peter M. Maurer and Peter M. Maurer. 1990. Generating test data with enhanced context-free grammars. IEEE Software, 7 (1990), 50–55.
[34]
Carlos Pacheco, Shuvendu Lahiri, Michael D. Ernst, and Tom Ball. 2006. Feedback-directed Random Test Generation. 14. https://www.microsoft.com/en-us/research/publication/ feedback-directed-random-test-generation/
[35]
Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In In ICSE. IEEE Computer Society.
[36]
Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. 2019. Semantic Fuzzing with Zest. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019). Association for Computing Machinery, New York, NY, USA. 329–340. isbn:9781450362245 https://doi.org/10.1145/3293882.3330576
[37]
Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, Carlos Noguera, and Lionel Seinturier. 2015. Spoon: A Library for Implementing Analyses and Transformations of Java Source Code. Software: Practice and Experience, 46 (2015), 1155–1179. https://doi.org/10.1002/spe.2346
[38]
Talia Ringer, Dan Grossman, Daniel Schwartz-Narbonne, and Serdar Tasiran. 2017. A Solver-aided Language for Test Input Generation. Proc. ACM Program. Lang., 1, OOPSLA (2017), Oct., 91:1–91:24. issn:2475-1421 https://doi.org/10.1145/3133915
[39]
Christoff Rossouw and Bernd Fischer. 2021. Vision: bias in systematic grammar-based test suite construction algorithms. In Proceedings of the 14th ACM SIGPLAN International Conference on Software Language Engineering. 143–149.
[40]
M. Sipser. 2012. Introduction to the Theory of Computation. Cengage Learning. isbn:9781285401065 https://books.google.co.uk/books?id=X0puCgAAQBAJ
[41]
Prashast Srivastava and Mathias Payer. 2021. Gramatron: Effective grammar-aware fuzzing. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 244–256.
[42]
L. D. Toffola, C. Staicu, and M. Pradel. 2017. Saying ‘Hi!’ is not enough: Mining inputs for effective test generation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 44–49. https://doi.org/10.1109/ASE.2017.8115617
[43]
Raja Vallee-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 2010. Soot: A Java bytecode optimization framework. In CASCON First Decade High Impact Papers. 214–224.
[44]
András Vargha and Harold D Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25, 2 (2000), 101–132.
[45]
Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2019. Superion: Grammar-aware Greybox Fuzzing. In Proceedings of the 41st International Conference on Software Engineering (ICSE ’19). IEEE Press, Piscataway, NJ, USA. 724–735. https://doi.org/10.1109/ICSE.2019.00081
[46]
Zhongxing Yu, Chenggang Bai, Lionel Seinturier, and Martin Monperrus. 2021. Characterizing the Usage, Evolution and Impact of Java Annotations in Practice. IEEE Transactions on Software Engineering, 47, 5 (2021), 969–986. https://doi.org/10.1109/TSE.2019.2910516
[47]
Michal Zalewski. 2007. American fuzzy lop. http://lcamtuf.coredump.cx/afl/
Index Terms
- June: A Type Testability Transformation for Improved ATG Performance
Recommendations
Achieving scalable mutation-based generation of whole test suites
Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...
Performance metamorphic testing: motivation and challenges
ICSE-NIER '17: Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results TrackPerformance testing is a challenging task mainly due to the lack of test oracles, that is, mechanisms to decide whether the performance of a program under a certain workload is either acceptable or poor due to a performance bug. Metamorphic testing ...
Refactoring as Testability Transformation
ICSTW '11: Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation WorkshopsThis paper briefly reviews the theory of Testability Transformation and outlines its implications for and relationship to refactoring for testing. The paper introduces testability refactorings, a subclass of Testability Transformations and discusses ...
Comments
Information & Contributors
Information
Published In
July 2023
1554 pages
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 July 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- EPSRC
Conference
ISSTA '23
Sponsor:
ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis
July 17 - 21, 2023
WA, Seattle, USA
Acceptance Rates
Overall Acceptance Rate 58 of 213 submissions, 27%
Upcoming Conference
ISSTA '25
- Sponsor:
- sigsoft
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 79Total Downloads
- Downloads (Last 12 months)31
- Downloads (Last 6 weeks)1
Reflects downloads up to 16 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in