research-article

Injecting mechanical faults to localize developer faults for evolving software

Authors:

Lingming Zhang,

Sarfraz KhurshidAuthors Info & Claims

OOPSLA '13: Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications

Pages 765 - 784

https://doi.org/10.1145/2509136.2509551

Published: 29 October 2013 Publication History

Abstract

This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based on the insight, we present the FIFL framework, which uses both the spectrum information of edits (obtained using the existing FaultTracer approach) as well as the potential impacts of edits (simulated by mutation changes) to achieve more accurate fault localization. We evaluate FIFL on real-world repositories of nine Java projects ranging from 5.7KLoC to 88.8KLoC. The experimental results show that FIFL is able to outperform the state-of-the-art FaultTracer technique for localizing failure-inducing program edits significantly. For example, all 19 FIFL strategies that use both the spectrum information and simulated impact information for each edit outperform the existing FaultTracer approach statistically at the significance level of 0.01. In addition, FIFL with its default settings outperforms FaultTracer by 2.33% to 86.26% on 16 of the 26 studied version pairs, and is only inferior than FaultTracer on one version pair.

References

[1]

R. Abreu, P. Zoeteweij, and A. J. Van Gemund. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference Practice and Research Techniques-MUTATION, pages 89--98, 2007.

Digital Library

[2]

E. Alves, M. Gligoric, V. Jagannath, and M. d'Amorim. Fault-localization using dynamic slicing and change impact analysis. In Proc. of ASE, pages 520--523, 2011.

Digital Library

[3]

P. Ammann and J. Offutt. Introduction to software testing. Cambridge University Press, 2008.

Digital Library

[4]

T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Theoretical and empirical studies on using program mutation to test the functional correctness of programs. In Proc. of POPL, pages 220--233, 1980.

Digital Library

[5]

O. Chesley, X. Ren, B. Ryder, and F. Tip. Crisp - A Fault Localization Tool for Java Programs. In Proc. of ICSE, pages 775--779, 2007.

Digital Library

[6]

B. Daniel, V. Jagannath, D. Dig, and D. Marinov. Reassert: Suggesting repairs for broken unit tests. In Proc. of ASE, pages 433--444, 2009.

Digital Library

[7]

B. Daniel, T. Gvero, and D. Marinov. On test repair using symbolic execution. In Proc. of ISSTA, pages 207--218, 2010.

Digital Library

[8]

R. DeMillo and A. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900--910, 1991.

Digital Library

[9]

R. DeMillo, R. Lipton, and F. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11 (4):34--41, 1978.

Digital Library

[10]

H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10 (4):405--435, 2005. ISSN 1382-3256.

Digital Library

[11]

P. G. Frankl, S. N. Weiss, and C. Hu. All-uses vs mutation testing: an experimental comparison of effectiveness. Journal of Systems and Software, 38(3):235--253, 1997.

Digital Library

[12]

G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. In Proc. of ISSTA, pages 147--158, 2010.

Digital Library

[13]

M. Gligoric, L. Zhang, C. Pereira, and G. Pokam. Selective mutation testing for concurrent code. In Proc. of ISSTA, pages 224--234, 2013.

Digital Library

[14]

R. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering, (4):279--290, 1977.

Digital Library

[15]

D. Hao, L. Zhang, Y. Pan, H. Mei, and J. Sun. On similarity-awareness in testing-based fault localization. Automated Software Engineering, 15(2):207--249, 2008.

Digital Library

[16]

D. Hao, L. Zhang, T. Xie, H. Mei, and J.-S. Sun. Interactive fault localization using test information. Journal of Computer Science and Technology, 24(5):962--974, 2009.

Digital Library

[17]

D. Hao, T. Xie, L. Zhang, X. Wang, J. Sun, and H. Mei. Test input reduction for result inspection to facilitate fault localization. Automated Software Engineering, 17(1):5--31, 2010.

Digital Library

[18]

M. Harman, Y. Jia, and W. Langdon. Strong higher order mutation-based test data generation. In Proc. of FSE, pages 212--222, 2011.

Digital Library

[19]

M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for Java software. In Proc. of OOPSLA, pages 312--326, 2001.

Digital Library

[20]

W. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering, (4):371--379, 1982.

Digital Library

[21]

R. Ihaka and R. Gentleman. R: A language for data analysis and graphics. Journal of computational and graphical statistics, 5(3):299--314, 1996.

[22]

D. Jeffrey, N. Gupta, and R. Gupta. Fault localization using value replacement. In Proc. of ISSTA, pages 167--178, 2008.

Digital Library

[23]

J. Jones, M. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proc. of ICSE, page 477, 2002.

Digital Library

[24]

S. Kim, E. J. Whitehead, and Y. Zhang. Classifying software changes: Clean or buggy? IEEE TSE, 34(2):181--196, 2008.

Digital Library

[25]

A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE TSE, 32(12):971--987, 2006.

Digital Library

[26]

E. Krauser, A. Mathur, and V. Rego. High performance software testing on simd machines. IEEE Transactions on Software Engineering, 17(5):403--423, 1991.

Digital Library

[27]

C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In Proc. of ICSE, pages 3--13, 2012.

Digital Library

[28]

B. Liblit, M. Naik, A. Zheng, A. Aiken, and M. Jordan. Scalable statistical bug isolation. In Proc. of PLDI, pages 15--26, 2005. ISBN 1595930566.

Digital Library

[29]

R. Lowry. Concepts and applications of inferential statistics. R. Lowry, 1998.

[30]

C. Mayer, S. Hanenberg, R. Robbes, É. Tanter, and A. Stefik. An empirical study of the influence of static type systems on the usability of undocumented software. In Proc. of OOPSLA, pages 683--702, 2012.

Digital Library

[31]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing junit test cases. TSE, 38 (6):1258--1275, 2012.

Digital Library

[32]

G. Misherghi and Z. Su. Hdd: hierarchical delta debugging. In Proc. of ICSE, pages 142--151, 2006.

Digital Library

[33]

E. Murphy-Hill, T. Zimmermann, C. Bird, and N. Nagappan. The design of bug fixes. In Proc. of ICSE, pages 332--341, 2013.

Digital Library

[34]

A. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology (TOSEM), 5(2):99--118, 1996.

Digital Library

[35]

A. Offutt, Z. Jin, and J. Pan. The dynamic domain reduction procedure for test data generation. Software-Practice and Experience, 29(2):167--194, 1999.

Digital Library

[36]

M. Papadakis and Y. L. Traon. Using mutants to locate unknown faults. In Proc. of ICST Workshop on Mutation Analysis, pages 691--700, 2012.

Digital Library

[37]

C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proc. of ISSTA, pages 199--209, 2011.

Digital Library

[38]

Y. Qi, X.Mao, Y. Lei, and C.Wang. Using automated program repair for evaluating the effectiveness of fault localization techniques. In Proc. of ISSTA, pages 191--201, 2013.

Digital Library

[39]

X. Ren and B. Ryder. Heuristic ranking of Java program edits for fault localization. In Proc. of ISSTA, pages 239--249, 2007.

Digital Library

[40]

X. Ren, F. Shah, F. Tip, B. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of Java programs. In Proc. of OOPSLA, 2004.

Digital Library

[41]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10):929--948, 2001.

Digital Library

[42]

D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. of FSE, pages 297--298, 2009.

Digital Library

[43]

D. Schuler, V. Dallmeier, and A. Zeller. Efficient mutation testing by checking invariant violations. In Proc. of ISSTA, pages 69--80, 2009.

Digital Library

[44]

S. S. Shapiro and M. B.Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591--611, 1965.

[45]

M. Stoerzer, B. Ryder, X. Ren, and F. Tip. Finding failure-inducing changes in Java programs using change classification. In Proc. of FSE, pages 57--68, 2006.

Digital Library

[46]

R. Tzoref, S. Ur, and E. Yom-Tov. Instrumenting where it hurts: an automatic concurrent debugging technique. In Proc. of ISSTA, pages 27--38, 2007.

Digital Library

[47]

R. Untch, A. Offutt, and M. Harrold. Mutation analysis using mutant schemata. In ACM SIGSOFT Software Engineering Notes, volume 18, pages 139--148, 1993.

Digital Library

[48]

W.Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In Proc. of ICSE, pages 364--374, 2009.

Digital Library

[49]

F. Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80--83, 1945.

[50]

G. Xu and A. Rountev. Regression test selection for aspectj software. In Proc. of ICSE, pages 65--74, 2007.

Digital Library

[51]

K. Yu, M. Lin, J. Chen, and X. Zhang. Practical isolation of failure-inducing changes for debugging regression faults. In Proc. of ASE, pages 20--29, 2012.

Digital Library

[52]

Y. Yu, J. Jones, and M. Harrold. An empirical study of the effects of test-suite reduction on fault localization. In Proc. of ICSE, pages 201--210, 2008.

Digital Library

[53]

A. Zeller. Yesterday, my program worked. today, it does not. why? In Proc. of FSE, pages 253--267, 1999.

Digital Library

[54]

A. Zeller. Automated debugging: Are we close? Computer, 34(11):26--31, 2001.

Digital Library

[55]

L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In Proc. of ICSE, pages 435--444, 2010.

Digital Library

[56]

L. Zhang, T. Xie, L. Zhang, N. Tillmann, J. De Halleux, and H. Mei. Test generation via dynamic symbolic execution for mutation testing. In Proc. of ICSM, pages 1--10, 2010.

Digital Library

[57]

L. Zhang, M. Kim, and S. Khurshid. Localizing failure-inducing program edits based on spectrum information. In Proc. of ICSM, pages 23--32, 2011.

Digital Library

[58]

L. Zhang, D.Marinov, L. Zhang, and S. Khurshid. Regression mutation testing. In Proc. of ISSTA, pages 331--341, 2012.

Digital Library

[59]

L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. Bridging the gap between the total and additional test-case prioritization strategies. In Proc. ICSE, pages 192--201, 2013.

Digital Library

[60]

L. Zhang, D. Marinov, and S. Khurshid. Faster mutation testing inspired by test prioritization and reduction. In Proc. of ISSTA, pages 235--245, 2013.

Digital Library

[61]

S. Zhang. Practical semantic test simplification. In Proc. of ICSE, pages 1173--1176, 2013.

Digital Library

[62]

S. Zhang, Y. Lin, Z. Gu, and J. Zhao. Effective identification of failure-inducing changes: a hybrid approach. In Proc. of PASTE, pages 77--83, 2008.

Digital Library

[63]

S. Zhang, C. Zhang, and M. D. Ernst. Automated documentation inference to explain failed tests. In Proc. of ASE, pages 63--72, 2011.

Digital Library

[64]

X. Zhang, N. Gupta, and R. Gupta. Locating faults through automated predicate switching. In Proc. of ICSE, pages 272--281, 2006.

Digital Library

[65]

X. Zhang, N. Gupta, and R. Gupta. Pruning dynamic slices with confidence. In Proc. of PLDI, pages 169--180, 2006.

Digital Library

[66]

T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy. Characterizing and predicting which bugs get reopened. In Proc. of ICSE, pages 1074--1083. IEEE, 2012.

Digital Library

Cited By

Li YWang SNguyen TRoychoudhury ACadar CKim M(2022)Fault localization to detect co-change fixing locationsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549137(659-671)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549137
Benton SLi XLou YZhang L(2022)Evaluating and Improving Unified DebuggingIEEE Transactions on Software Engineering10.1109/TSE.2021.312520348:11(4692-4716)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TSE.2021.3125203
Zhou ZJiang HRen ZChen YQiao L(2022)LocSeq: Automated Localization for Compiler Optimization Sequence Bugs of LLVMIEEE Transactions on Reliability10.1109/TR.2022.316537871:2(896-910)Online publication date: Jun-2022
https://doi.org/10.1109/TR.2022.3165378
Show More Cited By

Index Terms

Injecting mechanical faults to localize developer faults for evolving software
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Injecting mechanical faults to localize developer faults for evolving software
OOPSLA '13

This paper presents a novel methodology for localizing faults in code as it evolves. Our insight is that the essence of failure-inducing edits made by the developer can be captured using mechanical program transformations (e.g., mutation changes). Based ...
Are faults localizable?
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories

Many fault localization techniques have been proposed to facilitate debugging activities. Most of them attempt to pinpoint the location of faults (i.e., localize faults) based on a set of failing and correct executions and expect debuggers to ...
Injecting software faults in Python applications: The OpenStack case study
Abstract
Software fault injection techniques have been largely used as means for evaluating the dependability of systems in presence of certain types of faults. Despite the large diversity of tools offering the possibility of emulating the presence of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

OOPSLA '13: Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications

October 2013

904 pages

ISBN:9781450323741

DOI:10.1145/2509136

Co-chair:
Antony Hosking
Purdue University, USA
,
General Chair:
Patrick Eugster
Purdue University, USA
,
Program Chair:
Cristina V. Lopes
University of California, Irvine, USA

ACM SIGPLAN Notices Volume 48, Issue 10
OOPSLA '13
October 2013
867 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2544173
Editor:
Mark W. Bailey
Hamilton College, Clinton, NY
Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SPLASH '13

Sponsor:

SIGPLAN

SPLASH '13: Conference on Systems, Programming, and Applications: Software for Humanity

October 29 - 31, 2013

Indiana, Indianapolis, USA

Acceptance Rates

OOPSLA '13 Paper Acceptance Rate 50 of 189 submissions, 26%;

Overall Acceptance Rate 268 of 1,244 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

67
Total Citations
View Citations
342
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)3

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li YWang SNguyen TRoychoudhury ACadar CKim M(2022)Fault localization to detect co-change fixing locationsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549137(659-671)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549137
Benton SLi XLou YZhang L(2022)Evaluating and Improving Unified DebuggingIEEE Transactions on Software Engineering10.1109/TSE.2021.312520348:11(4692-4716)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TSE.2021.3125203
Zhou ZJiang HRen ZChen YQiao L(2022)LocSeq: Automated Localization for Compiler Optimization Sequence Bugs of LLVMIEEE Transactions on Reliability10.1109/TR.2022.316537871:2(896-910)Online publication date: Jun-2022
https://doi.org/10.1109/TR.2022.3165378
Ding YSuneja SZheng YLaredo JMorari AKaiser GRay B(2022)VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00114(959-970)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00114
Zhang MLi YLi XChen LZhang YZhang LKhurshid S(2021)An Empirical Study of Boosting Spectrum-Based Fault Localization via PageRankIEEE Transactions on Software Engineering10.1109/TSE.2019.291128347:6(1089-1113)Online publication date: 1-Jun-2021
https://doi.org/10.1109/TSE.2019.2911283
Li XJiang JBenton SXiong YZhang L(2021)A Large-scale Study on API Misuses in the Wild2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST49551.2021.00034(241-252)Online publication date: Apr-2021
https://doi.org/10.1109/ICST49551.2021.00034
Li YWang SNguyen T(2021)Fault Localization with Code Coverage Representation LearningProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00067(661-673)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00067
Chen JMa HZhang LGrundy JLe Goues CLo D(2020)Enhanced compiler bug isolation via memoized searchProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416570(78-89)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416570
Kim YMun SYoo SKim M(2019)Precise Learn-to-Rank Fault Localization Using Dynamic and Static Features of Target ProgramsACM Transactions on Software Engineering and Methodology10.1145/334562828:4(1-34)Online publication date: 9-Oct-2019
https://dl.acm.org/doi/10.1145/3345628
Chen JHan JSun PZhang LHao DZhang LDumas MPfahl DApel SRusso A(2019)Compiler bug isolation via effective witness test program generationProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338957(223-234)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338957
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten