research-article

Defects4J: a database of existing faults to enable controlled testing studies for Java programs

Authors:

René Just,

Darioush Jalali,

Michael D. ErnstAuthors Info & Claims

ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

Pages 437 - 440

https://doi.org/10.1145/2610384.2628055

Published: 21 July 2014 Publication History

Get Access

Abstract

Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.

References

[1]

V. Dallmeier and T. Zimmermann. Extraction of bug localization benchmarks from history. In Proceedings of the International Conference on Automated Software Engineering (ASE), pages 433–436, 2007.

Digital Library

Google Scholar

[2]

H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10(4):405–435, 2005.

Digital Library

Google Scholar

[3]

G. Fraser and A. Arcuri. Evosuite: Automatic test suite generation for object-oriented software. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 416–419, 2011.

Digital Library

Google Scholar

[4]

M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the International Conference on Software Engineering (ICSE), pages 191–200, 1994.

Digital Library

Google Scholar

[5]

R. Just. The Major mutation framework: Eﬃcient and scalable mutation analysis for Java. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2014. To appear.

Digital Library

Google Scholar

[6]

R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? Technical Report UW-CSE-14-02-02, University of Washington, 2014.

Digital Library

Google Scholar

[7]

R. Just, G. M. Kapfhammer, and F. Schweiggert. Using non-redundant mutation operators and test suite prioritization to achieve eﬃcient and scalable mutation analysis. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE), pages 11–20, 2012.

Digital Library

Google Scholar

[8]

A. J. Ko and B. A. Myers. A framework and methodology for studying the causes of software errors in programming systems. Journal of Visual Languages & Computing, 16(1):41––84, 2005.

Digital Library

Google Scholar

Cited By

View all

Wu WWang SLiu B(2024)Software Fault Localization Based on Weighted Association Rule Mining and Complex NetworksMathematics10.3390/math1213211312:13(2113)Online publication date: 5-Jul-2024
https://doi.org/10.3390/math12132113
Hao SShi XLiu H(2024)Exploring the Potential of Pre-Trained Language Models of Code for Automated Program RepairElectronics10.3390/electronics1307120013:7(1200)Online publication date: 25-Mar-2024
https://doi.org/10.3390/electronics13071200
Tan YTian J(2024)A Method for Processing Static Analysis Alarms Based on Deep LearningApplied Sciences10.3390/app1413554214:13(5542)Online publication date: 26-Jun-2024
https://doi.org/10.3390/app14135542
Show More Cited By

Index Terms

Defects4J: a database of existing faults to enable controlled testing studies for Java programs
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid ...
Searching for Multi-fault Programs in Defects4J
Search-Based Software Engineering
Abstract
Defects4J has enabled numerous software testing and debugging research work since its introduction. A large part of its contribution, and the resulting popularity, lies in the clear separation and distillation of the root cause of each individual ...
GitBug-Java: A Reproducible Java Benchmark of Recent Bugs
MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

Bug-fix benchmarks are essential for evaluating methodologies in automatic program repair (APR) and fault localization (FL). However, existing benchmarks, exemplified by Defects4J, need to evolve to incorporate recent bug-fixes aligned with contemporary ...

Comments

Information & Contributors

Information

Published In

ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

July 2014

460 pages

ISBN:9781450326452

DOI:10.1145/2610384

General Chair:
Corina S. Păsăreanu
NASA Ames, USA
,
Program Chair:
Darko Marinov
University of Illinois at Urbana-Champaign, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '14

Sponsor:

SIGSOFT

ISSTA '14: International Symposium on Software Testing and Analysis

July 21 - 25, 2014

CA, San Jose, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

799
Total Citations
View Citations
2,587
Total Downloads

Downloads (Last 12 months)492
Downloads (Last 6 weeks)42

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu WWang SLiu B(2024)Software Fault Localization Based on Weighted Association Rule Mining and Complex NetworksMathematics10.3390/math1213211312:13(2113)Online publication date: 5-Jul-2024
https://doi.org/10.3390/math12132113
Hao SShi XLiu H(2024)Exploring the Potential of Pre-Trained Language Models of Code for Automated Program RepairElectronics10.3390/electronics1307120013:7(1200)Online publication date: 25-Mar-2024
https://doi.org/10.3390/electronics13071200
Tan YTian J(2024)A Method for Processing Static Analysis Alarms Based on Deep LearningApplied Sciences10.3390/app1413554214:13(5542)Online publication date: 26-Jun-2024
https://doi.org/10.3390/app14135542
Cao HHan DChu YTian FWang YLiu YJia JGe H(2024)Multi-mechanism neural machine translation framework for automatic program repairJournal of Intelligent & Fuzzy Systems10.3233/JIFS-23403746:4(7859-7873)Online publication date: 18-Apr-2024
https://doi.org/10.3233/JIFS-234037
Li YPei HHuang LYin BCai K(2024)Automatic Repair of Quantum Programs via Unitary OperationACM Transactions on Software Engineering and Methodology10.1145/366460433:6(1-43)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664604
Rafi Md'Amorim M(2024)Enhancing Code Representation for Improved Graph Neural Network-Based Fault LocalizationCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3664459(686-688)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3664459
Wu YLi ZZhang JLiu Yd'Amorim M(2024)ConDefects: A Complementary Dataset to Address the Data Leakage Concern for LLM-Based Fault Localization and Program RepairCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663815(642-646)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663815
Smytzek MEberlein MSerçe BGrunske LZeller Ad'Amorim M(2024)Tests4Py: A Benchmark for System TestingCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663798(557-561)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663798
Petke JMartinez MKechagia MAleti ASarro Fd'Amorim M(2024)The Patch Overfitting Problem in Automated Program Repair: Practical Magnitude and a Baseline for Realistic BenchmarkingCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663776(452-456)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663776
Ságodi ZAntal GBogenfürst BIsztin MHegedűs PFerenc R(2024)Reality Check: Assessing GPT-4 in Fixing Real-World Software VulnerabilitiesProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661207(252-261)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661207
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies

Searching for Multi-fault Programs in Defects4J

GitBug-Java: A Reproducible Java Benchmark of Recent Bugs

Comments

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies

Searching for Multi-fault Programs in Defects4J

GitBug-Java: A Reproducible Java Benchmark of Recent Bugs

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations