TestSpark: IntelliJ IDEA's Ultimate Test Generation Companion
Authors: 
Arkadii Sapozhnikov, 
Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, Pouria DerakhshanfarAuthors Info & Claims
Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, Pouria DerakhshanfarAuthors Info & ClaimsPages 30 - 34
Abstract
Writing software tests is laborious and time-consuming. To address this, prior studies introduced various automated test-generation techniques. A well-explored research direction in this field is unit test generation, wherein artificial intelligence (AI) techniques create tests for a method/class under test. While many of these techniques have primarily found applications in a research context, existing tools (e.g., EvoSuite, Randoop, and AthenaTest) are not user-friendly and are tailored to a single technique. This paper introduces Test-Spark, a plugin for IntelliJ IDEA that enables users to generate unit tests with only a few clicks directly within their Integrated Development Environment (IDE). Furthermore, TestSpark also allows users to easily modify and run each generated test and integrate them into the project workflow. TestSpark leverages the advances of search-based test generation tools, and it introduces a technique to generate unit tests using Large Language Models (LLMs) by creating a feedback cycle between the IDE and the LLM. Since TestSpark is an open-source (https://github.com/JetBrains-Research/TestSpark), extendable, and well-documented tool, it is possible to add new test generation methods into the plugin with the minimum effort. This paper also explains our future studies related to TestSpark and our preliminary results. Demo video: https://youtu.be/0F4PrxWfiXo
References
[1]
2021. Kex. https://github.com/vorpal-research/kex/tree/sbst-contest.
[2]
2022. Kex-Reflection. https://github.com/vorpal-research/kex/tree/sbst2022-reflection.
[3]
2022. UTBot. https://github.com/UnitTestBot.
[4]
Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Soft. Eng. and Methodology (TOSEM) 28, 1 (2019), 1--37.
[5]
Patrick Bareiß, Beatriz Souza, Marcelo d'Amorim, and Michael Pradel. 2022. Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code. CoRR abs/2206.01335 (2022).
[6]
Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman. 2015. When, how, and why developers (do not) test in their IDEs. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 179--190.
[7]
Pietro Braione, Giovanni Denaro, Andrea Mattavelli, and Mauro Pezzé. 2018. SUSHI: a test generator for programs with complex structured inputs. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. 21--24.
[8]
Mariano Ceccato, Alessandro Marchetto, Leonardo Mariani, Cu D Nguyen, and Paolo Tonella. 2015. Do automatically generated test cases make debugging easier? an experimental assessment of debugging effectiveness and efficiency. ACM Transactions on Soft. Eng. and Methodology (TOSEM) 25, 1 (2015), 1--38.
[9]
Pouria Derakhshanfar and Xavier Devroey. 2022. Basic block coverage for unit test generation at the SBST 2022 tool competition. In Proceedings of the 15th Workshop on Search-Based Software Testing. 37--38.
[10]
Pouria Derakhshanfar, Xavier Devroey, Annibale Panichella, Andy Zaidman, and Arie van Deursen. 2022. Generating Class-Level Integration Tests Using Call Site Information. IEEE Transactions on Software Engineering 49, 4 (2022), 2069--2087.
[11]
Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416--419.
[12]
Gunel Jahangirova and Valerio Terragni. 2023. SBFT tool competition 2023-Java test case generation track. In 2023 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT). IEEE, 61--64.
[13]
Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2312--2323.
[14]
Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping coverage plateaus in test generation with pre-trained large language models. In International conference on software engineering (ICSE).
[15]
Phil McMinn. 2011. Search-based software testing: Past, present and future. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 153--163.
[16]
Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2017. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering 44, 2 (2017), 122--158.
[17]
Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2018. A large scale empirical comparison of state-of-the-art search-based test case generators. Information and Software Technology 104 (2018), 236--256.
[18]
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering (2023).
[19]
Sina Shamshiri, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In 2015 30th IEEE/ACM Int. Conference on Automated Software Engineering (ASE). IEEE, 201--211.
[20]
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020).
Index Terms
- TestSpark: IntelliJ IDEA's Ultimate Test Generation Companion
Recommendations
HITS: High-coverage LLM-based Unit Test Generation via Method Slicing
ASE '24: Proceedings of the 39th IEEE/ACM International Conference on Automated Software EngineeringLarge language models (LLMs) have behaved well in generating unit tests for Java projects. However, the performance for covering the complex focal methods within the projects is poor. Complex methods comprise many conditions and loops, requiring the test ...
ChatUniTest: A Framework for LLM-Based Test Generation
FSE 2024: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software EngineeringUnit testing is an essential yet frequently arduous task. Various automated unit test generation tools have been introduced to mitigate this challenge. Notably, methods based on large language models (LLMs) have garnered considerable attention and ...
Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools
LLM4Code '24: Proceedings of the 1st International Workshop on Large Language Models for CodeGenerating unit tests is a crucial task in software development, demanding substantial time and effort from programmers. The advent of Large Language Models (LLMs) introduces a novel avenue for unit test script generation. This research aims to ...
Comments
Information & Contributors
Information
Published In
April 2024
531 pages
ISBN:9798400705021
DOI:10.1145/3639478
- Co-chairs:
- Ana Paiva,
- Rui Abreu,
- Program Co-chairs:
- Abhik Roychoudhury,
- Margaret Storey
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- Faculty of Engineering of University of Porto
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 23 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICSE-Companion '24
Sponsor:
ICSE-Companion '24: 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings
April 14 - 20, 2024
Lisbon, Portugal
Acceptance Rates
Overall Acceptance Rate 276 of 1,856 submissions, 15%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 148Total Downloads
- Downloads (Last 12 months)148
- Downloads (Last 6 weeks)26
Reflects downloads up to 17 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in