research-article

Is the cure worse than the disease? overfitting in automated program repair

Authors:

Edward K. Smith,

Claire Le Goues,

Yuriy BrunAuthors Info & Claims

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Pages 532 - 543

https://doi.org/10.1145/2786805.2786825

Published: 30 August 2015 Publication History

Abstract

Automated program repair has shown promise for reducing the significant manual effort debugging requires. This paper addresses a deficit of earlier evaluations of automated repair techniques caused by repairing programs and evaluating generated patches' correctness using the same set of tests. Since tests are an imperfect metric of program correctness, evaluations of this type do not discriminate between correct patches and patches that overfit the available tests and break untested but desired functionality. This paper evaluates two well-studied repair tools, GenProg and TrpAutoRepair, on a publicly available benchmark of bugs, each with a human-written patch. By evaluating patches using tests independent from those used during repair, we find that the tools are unlikely to improve the proportion of independent tests passed, and that the quality of the patches is proportional to the coverage of the test suite used during repair. For programs that pass most tests, the tools are as likely to break tests as to fix them. However, novice developers also overfit, and automated repair performs no worse than these developers. In addition to overfitting, we measure the effects of test suite coverage, test suite provenance, and starting program quality, as well as the difference in quality between novice-developer-written and tool-generated patches when quality is assessed with a test suite independent from the one used for patch generation.

References

[1]

M. Abd-El-Malek, G. R. Ganger, G. R. Goodson, M. K. Reiter, and J. J. Wylie. Fault-scalable Byzantine fault-tolerant services. In ACM Symposium on Operating Systems Principles (SOSP), pages 59–74, Brighton, UK, 2005.

Digital Library

[2]

R. E. Adamson. Functional fixedness as related to problem solving: A repetition of three experiments. Journal of Experimental Psychology, 44(4):288–291, 1952.

[3]

E. Alba and F. Chicano. Finding safety errors with ACO. In Conference on Genetic and Evolutionary Computation (GECCO), pages 1066–1073, London, England, UK, July 2007.

Digital Library

[4]

M. Alkhalaf, A. Aydin, and T. Bultan. Semantic differential repair for input validation and sanitization. In International Symposium on Software Testing and Analysis (ISSTA), pages 225–236, San Jose, CA, USA, July 2014.

Digital Library

[5]

A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 1–10, Honolulu, HI, USA, 2011.

Digital Library

[6]

A. Arcuri and X. Yao. A novel co-evolutionary approach to automatic software bug fixing. In Congress on Evolutionary Computation, pages 162–168, 2008.

[7]

E. T. Barr, Y. Brun, P. Devanbu, M. Harman, and F. Sarro. The plastic surgery hypothesis. In Symposium on the Foundations of Software Engineering (FSE), pages 306–317, Hong Kong, China, November 2014.

Digital Library

[8]

A. Barreto, M. Barros, and C. Werner. Staffing a software project: A constraint satisfaction approach. Computers and Operations Research, 35(10):3073–3089, 2008.

Digital Library

[9]

A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros, A. Kamsky, S. McPeak, and D. Engler. A few billion lines of code later: Using static analysis to find bugs in the real world. Communications of the ACM, 53(2):66–75, Feb. 2010.

Digital Library

[10]

C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu. Fair and balanced?: Bias in bugfix datasets. In European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE), pages 121–130, Amsterdam, The Netherlands, August 2009.

Digital Library

[11]

Y. Brun, E. Barr, M. Xiao, C. Le Goues, and P. Devanbu. Evolution vs. intelligent design in program patching. Technical Report https://escholarship.org/uc/item/3z8926ks, UC Davis: College of Engineering, 2013.

[12]

C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In USENIX Conference on Operating Systems Design and Implementation (OSDI), pages 209–224, San Diego, CA, USA, 2008.

Digital Library

[13]

A. Carzaniga, A. Gorla, A. Mattavelli, N. Perino, and M. Pezzè. Automatic recovery from runtime failures. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 782–791, San Francisco, CA, USA, 2013.

Digital Library

[14]

A. Carzaniga, A. Gorla, N. Perino, and M. Pezzè. Automatic workarounds for web applications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pages 237–246, Santa Fe, New Mexico, USA, 2010.

Digital Library

[15]

L. Chen and A. Avižienis. N-version programming: A faulttolerance approach to reliability of software operation. In IEEE International Symposium on Fault-Tolerant Computing (FTCS), pages 3–9, 1978.

[16]

R. Cochran, L. D’Antoni, B. Livshits, D. Molnar, and M. Veanes. Program boosting: Program synthesis via crowdsourcing. In Symposium on Principles of Programming Languages (POPL), pages 677–688, Mumbai, India, January 2015.

Digital Library

[17]

Z. Coker and M. Hafiz. Program transformations to fix C integers. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 792–801, San Francisco, CA, USA, 2013.

Digital Library

[18]

V. Debroy and W. Wong. Using mutation to automatically suggest fixes for faulty programs. In International Conference on Software Testing, Verification, and Validation, pages 65–74, Paris, France, 2010.

Digital Library

[19]

T. Durieux, M. Martinez, M. Monperrus, R. Sommerard, and J. Xuan. Automatic repair of real bugs: An experience report on the Defects4J dataset. CoRR, abs/1505.07002, 2015.

[20]

H.-C. Estler, C. A. Furia, M. Nordio, M. Piccioni, and B. Meyer. Contracts in practice. In International Symposium on Formal Methods (FM), pages 230–246, Singapore, May 2014.

Digital Library

[21]

Z. P. Fry, B. Landau, and W. Weimer. A human study of patch maintainability. In International Symposium on Software Testing and Analysis (ISSTA), pages 177–187, Minneapolis, MN, USA, July 2012.

Digital Library

[22]

M. Gabel and Z. Su. Testing mined specifications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), Cary, NC, USA, 2012.

Digital Library

[23]

S. Gulwani. Automating string processing in spreadsheets using input-output examples. In Symposium on Principles of Programming Languages (POPL), pages 317–330, Austin, TX, USA, 2011.

Digital Library

[24]

S. Gustafson, A. Ekárt, E. Burke, and G. Kendall. Problem difficulty and code growth in genetic programming. Genetic Programming and Evolvable Machines, pages 271–290, Sept. 2004.

Digital Library

[25]

M. Harman. The current state and future of search based software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 342–357, 2007.

Digital Library

[26]

G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit. Automated atomicity-violation fixing. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 389–400, San Jose, CA, USA, 2011.

Digital Library

[27]

R. Just, D. Jalali, and M. D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pages 437–440, San Jose, CA, USA, July 2014.

Digital Library

[28]

Y. Ke, K. T. Stolee, C. Le Goues, and Y. Brun. Repairing programs with semantic code search. In Proceedings of the 30th IEEE/ACM International Conference On Automated Software Engineering (ASE), Lincoln, NE, USA, November 2015.

Digital Library

[29]

D. Kim, J. Nam, J. Song, and S. Kim. Automatic patch generation learned from human-written patches. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 802–811, San Francisco, CA, USA, 2013.

Digital Library

[30]

J. C. Knight and N. G. Leveson. An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering (TSE), 12(1):96– 109, 1986.

Digital Library

[31]

J. R. Koza. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, 1992.

Digital Library

[32]

C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In AMC/IEEE International Conference on Software Engineering (ICSE), pages 3–13, Zurich, Switzerland, 2012.

Digital Library

[33]

C. Le Goues, S. Forrest, and W. Weimer. Representations and operators for improving evolutionary software repair. In Conference on Genetic and Evolutionary Computation (GECCO), pages 959–966, Philadelphia, PA, USA, July 2012.

Digital Library

[34]

C. Le Goues, N. Holtschulte, E. K. Smith, Y. Brun, P. Devanbu, S. Forrest, and W. Weimer. The ManyBugs and Intro-Class benchmarks for automated repair of C programs. IEEE Transactions on Software Engineering (TSE), in press, 2015.

[35]

C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. Gen-Prog: A generic method for automatic software repair. IEEE Transactions on Software Engineering (TSE), 38:54–72, 2012.

Digital Library

[36]

Y. Lin and S. S. Kulkarni. Automatic repair for multi-threaded programs with deadlock/livelock using maximum satisfiability. In International Symposium on Software Testing and Analysis (ISSTA), pages 237–247, San Jose, CA, USA, July 2014.

Digital Library

[37]

P. Liu, O. Tripp, and C. Zhang. Grail: Context-aware fixing of concurrency bugs. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pages 318– 329, Hong Kong, China, Nov. 2014.

Digital Library

[38]

A. S. Luchins. Mechanization in problem solving: The effect of Einstellung. Psychological Monographs, 54(6):i–95, 1942.

[39]

S. Mechtaev, J. Yi, and A. Roychoudhury. DirectFix: Looking for simple program repairs. In International Conference on Software Engineering (ICSE), Florence, Italy, May 2015.

Digital Library

[40]

C. C. Michael, G. McGraw, and M. A. Schatz. Generating software test data by evolution. IEEE Transactions on Software Engineering (TSE), 27(12):1085–1110, Dec. 2001.

Digital Library

[41]

T. M. Mitchell. Machine Learning. McGraw-Hill, New York, 1997.

Digital Library

[42]

M. Monperrus. A critical review of “Automatic patch generation learned from human-written patches”: Essay on the problem statement and the evaluation of automatic software repair. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 234–242, Hyderabad, India, June 2014.

Digital Library

[43]

H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra. SemFix: Program repair via semantic analysis. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 772–781, San Francisco, CA, USA, 2013.

Digital Library

[44]

M. Orlov and M. Sipper. Flight of the FINCH through the Java wilderness. IEEE Transactions on Evolutionary Computation, 15(2):166–182, Apr. 2011.

Digital Library

[45]

C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In International Symposium on Software Testing and Analysis (ISSTA), pages 199–209, Toronto, ON, Canada, 2011.

Digital Library

[46]

Y. Pei, C. A. Furia, M. Nordio, Y. Wei, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. IEEE Transactions on Software Engineering (TSE), 40(5):427–449, 2014.

Digital Library

[47]

J. H. Perkins, S. Kim, S. Larsen, S. Amarasinghe, J. Bachrach, M. Carbin, C. Pacheco, F. Sherwood, S. Sidiroglou, G. Sullivan, W.-F. Wong, Y. Zibin, M. D. Ernst, and M. Rinard. Automatically patching errors in deployed software. In ACM Symposium on Operating Systems Principles (SOSP), pages 87–102, Big Sky, MT, USA, October 12–14, 2009.

Digital Library

[48]

D. Posnett, V. Filkov, and P. Devanbu. Ecological inference in empirical software engineering. In International Conference on Automated Software Engineering (ASE), pages 362–371, Lawrence, KS, USA, November 2011.

Digital Library

[49]

Y. Qi, X. Mao, and Y. Lei. Efficient automated program repair through fault-recorded testing prioritization. In International Conference on Software Maintenance (ICSM), pages 180–189, Eindhoven, The Netherlands, Sept. 2013.

Digital Library

[50]

Z. Qi, F. Long, S. Achour, and M. Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In International Symposium on Software Testing and Analysis (ISSTA), pages 24–36, Baltimore, MD, USA, 2015.

Digital Library

[51]

Research Triangle Institute. The economic impacts of inadequate infrastructure for software testing. NIST Planning Report 02-3, May 2002.

[52]

J. Rissanen. Modelling by the shortest data description. Automatica, 14:465–471, 1978.

Digital Library

[53]

O. Seng, J. Stammel, and D. Burkhart. Search-based determination of refactorings for improving the class structure of object-oriented systems. In Conference on Genetic and Evolutionary Computation (GECCO), pages 1909–1916, Seattle, WA, USA, July 2006.

Digital Library

[54]

S. Sidiroglou and A. D. Keromytis. Countering network worms through automatic patch generation. IEEE Security and Privacy, 3(6):41–49, Nov. 2005.

Digital Library

[55]

S. Sidiroglou-Douskos, E. Lahtinen, F. Long, and M. Rinard. Automatic error elimination by horizontal code transfer across multiple applications. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 43–54, Portland, OR, USA, 2015.

Digital Library

[56]

S. Silva and E. Costa. Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories. Genetic Programming and Evolvable Machines, 10(2):141–179, June 2009.

Digital Library

[57]

A. Smirnov and T. cker Chiueh. Dira: Automatic detection, identification and repair of control-hijacking attacks. In Network and Distributed System Security Symposium (NDSS), San Diego, CA, USA, Feb. 2005.

[58]

S. H. Tan and A. Roychoudhury. relifix: Automated repair of software regressions. In International Conference on Software Engineering (ICSE), Florence, Italy, May 2015.

[59]

K. R. Walcott, M. Soffa, G. M. Kapfhammer, and R. S. Roos. Time-aware test suite prioritization. In International Symposium on Software Testing and Analysis (ISSTA), pages 1–12, Portland, ME, USA, July 2006.

Digital Library

[60]

Y. Wei, Y. Pei, C. A. Furia, L. S. Silva, S. Buchholz, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. In International Symposium on Software Testing and Analysis (ISSTA), pages 61–72, Trento, Italy, 2010.

Digital Library

[61]

W. Weimer, Z. P. Fry, and S. Forrest. Leveraging program equivalence for adaptive program repair: Models and first results. In IEEE/ACM International Conference on Automated Software Engineering (ASE), Palo Alto, CA, USA, 2013.

Digital Library

[62]

W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 364–374, Vancouver, BC, Canada, 2009.

Digital Library

[63]

A. Zeller and R. Hildebrandt. Simplifying and isolating failureinducing input. IEEE Transactions on Software Engineering, 28(2):183–200, February 2002.

Digital Library

[64]

H. Zhong and Z. Su. An empirical study on real bug fixes. In ACM/IEEE International Conference on Software Engineering (ICSE), Florence, Italy, May 2015.

Cited By

Petke JMartinez MKechagia MAleti ASarro Fd'Amorim M(2024)The Patch Overfitting Problem in Automated Program Repair: Practical Magnitude and a Baseline for Realistic BenchmarkingCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663776(452-456)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663776
Williams DCallan JKirbas SMechtaev SPetke JPrideaux-Ghee TSarro FRoychoudhury APaiva AAbreu RStorey MAniche MNagappan N(2024)User-Centric Deployment of Automated Program Repair at BloombergProceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice10.1145/3639477.3639756(81-91)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639477.3639756
Yang WSong LXue YRoychoudhury APaiva AAbreu RStorey M(2024)Rust-lancet: Automated Ownership-Rule-Violation Fixing with Behavior PreservationProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639103(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639103
Show More Cited By

Index Terms

Is the cure worse than the disease? overfitting in automated program repair
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

A correlation study between automated program repair and test-suite metrics

Automated program repair is increasingly gaining traction, due to its potential to reduce debugging cost greatly. The feasibility of automated program repair has been shown in a number of works, and the research focus is gradually shifting toward the ...
Efficient Automated Program Repair through Fault-Recorded Testing Prioritization
ICSM '13: Proceedings of the 2013 IEEE International Conference on Software Maintenance

Most techniques for automated program repair use test cases to validate the effectiveness of the produced patches. The validation process can be time-consuming especially when the object programs ship with either lots of test cases or some long-running ...
Do automated program repair techniques repair hard and important bugs?

Existing evaluations of automated repair techniques focus on the fraction of the defects for which the technique can produce a patch, the time needed to produce patches, and how well patches generalize to the intended specification. However, these ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

August 2015

1068 pages

ISBN:9781450336758

DOI:10.1145/2786805

General Chair:
Elisabetta Di Nitto
Politecnico di Milano, Italy
,
Program Chairs:
Mark Harman
University College London, UK
,
Patrick Heymans
University of Namur, Belgium

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ESEC/FSE'15

Sponsor:

SIGSOFT

ESEC/FSE'15: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering

August 30 - September 4, 2015

Bergamo, Italy

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

216
Total Citations
View Citations
1,121
Total Downloads

Downloads (Last 12 months)88
Downloads (Last 6 weeks)4

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Petke JMartinez MKechagia MAleti ASarro Fd'Amorim M(2024)The Patch Overfitting Problem in Automated Program Repair: Practical Magnitude and a Baseline for Realistic BenchmarkingCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663776(452-456)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663776
Williams DCallan JKirbas SMechtaev SPetke JPrideaux-Ghee TSarro FRoychoudhury APaiva AAbreu RStorey MAniche MNagappan N(2024)User-Centric Deployment of Automated Program Repair at BloombergProceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice10.1145/3639477.3639756(81-91)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639477.3639756
Yang WSong LXue YRoychoudhury APaiva AAbreu RStorey M(2024)Rust-lancet: Automated Ownership-Rule-Violation Fixing with Behavior PreservationProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639103(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639103
Eladawy HLe Goues CBrun YRoychoudhury APaiva AAbreu RStorey M(2024)Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639095(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639095
Zhong WLi CLiu KXu TGe JBissyande TLuo BNg VRoychoudhury APaiva AAbreu RStorey M(2024)Practical Program Repair via Preference-based Ensemble StrategyProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623310(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623310
Zhang QFang CSun WLiu YHe THao XChen Z(2024)APPT: Boosting Automated Patch Correctness Prediction via Fine-Tuning Pre-Trained ModelsIEEE Transactions on Software Engineering10.1109/TSE.2024.335496950:3(474-494)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3354969
Yang DHe JMao XLi TLei YYi XWu J(2024)Strider: Signal Value Transition-Guided Defect Repair for HDL Programming AssignmentsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.334175043:5(1594-1607)Online publication date: May-2024
https://doi.org/10.1109/TCAD.2023.3341750
Li HRajbahadur GLin DBezemer CJiang Z(2024)Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate OverfittingIEEE Access10.1109/ACCESS.2024.340254312(70676-70689)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3402543
Dong YCheng XYang YZhang LWang SKong L(2024)A Method to Identify Overfitting Program Repair Patches Based on Expression TreeScience of Computer Programming10.1016/j.scico.2024.103105(103105)Online publication date: Mar-2024
https://doi.org/10.1016/j.scico.2024.103105
Gharibi RSadreddini MFakhrahmad S(2024)T5APRJournal of Systems and Software10.1016/j.jss.2024.112083214:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112083
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents