research-article

Interactive metamorphic testing of debuggers

Authors:

Sandro Tolksdorf,

Daniel Lehmann,

Michael PradelAuthors Info & Claims

ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 273 - 283

https://doi.org/10.1145/3293882.3330567

Published: 10 July 2019 Publication History

Abstract

When improving their code, developers often turn to interactive debuggers. The correctness of these tools is crucial, because bugs in the debugger itself may mislead a developer, e.g., to believe that executed code is never reached or that a variable has another value than in the actual execution. Yet, debuggers are difficult to test because their input consists of both source code and a sequence of debugging actions, such as setting breakpoints or stepping through code. This paper presents the first metamorphic testing approach for debuggers. The key idea is to transform both the debugged code and the debugging actions in such a way that the behavior of the original and the transformed inputs should differ only in specific ways. For example, adding a breakpoint should not change the control flow of the debugged program. To support the interactive nature of debuggers, we introduce interactive metamorphic testing. It differs from traditional metamorphic testing by determining the input transformation and the expected behavioral change it causes while the program under test is running. Our evaluation applies the approach to the widely used debugger in the Chromium browser, where it finds eight previously unknown bugs with a true positive rate of 51%. All bugs have been confirmed by the developers, and one bug has even been marked as release-blocking.

References

[1]

Shay Artzi, Julian Dolby, Simon Holm Jensen, Anders Møller, and Frank Tip. 2011. A framework for automated testing of JavaScript web applications. In ICSE. 571–580.

Digital Library

[2]

Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (2015), 507–525.

Digital Library

[3]

Alexandra Bugariu, Valentin WÃĳstholz, Maria Christakis, and Peter MÃĳller. 2018. Automatically Testing Implementations of Numerical Abstract Domains. In ASE.

Digital Library

[4]

Cristian Cadar and Alastair F Donaldson. 2016. Analysing the program analyser. In Proceedings of the 38th International Conference on Software Engineering Companion. ACM, 765–768.

Digital Library

[5]

Wing Kwong Chan, Shing Chi Cheung, and Karl RPH Leung. 2007. A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research (IJWSR) 4, 2 (2007), 61–81.

[6]

Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An Empirical Comparison of Compiler Testing Techniques. In ICSE.

Digital Library

[7]

Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong âĂę.

[8]

Tsong Yueh Chen, Jianqiang Feng, and T. H. Tse. 2002. Metamorphic Testing of Programs on Partial Differential Equations: A Case Study. In 26th International Computer Software and Applications Conference (COMPSAC 2002), Prolonging Software Life: Development and Redevelopment, 26-29 August 2002, Oxford, England, Proceedings. 327–333.

Digital Library

[9]

Tsong Yueh Chen, DH Huang, TH Tse, and Zhi Quan Zhou. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004). Polytechnic University of Madrid, 569–583.

[10]

Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming compiler fuzzers. In Conference on Programming Language Design and Implementation (PLDI). 197–208.

Digital Library

[11]

Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated Testing of Refactoring Engines. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC-FSE ’07). ACM, New York, NY, USA, 185–194.

Digital Library

[12]

Alastair F. Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated Testing of Graphics Shader Compilers. Proc. ACM Program. Lang. 1, OOPSLA, Article 93 (Oct. 2017), 29 pages.

Digital Library

[13]

Markus Ermuth and Michael Pradel. 2016. Monkey See, Monkey Do: Effective Generation of GUI Tests with Inferred Macro Events. In International Symposium on Software Testing and Analysis (ISSTA). 82–93.

Digital Library

[14]

Satia Herfert, Jibesh Patra, and Michael Pradel. 2017. Automatically Reducing Tree-Structured Test Inputs. In ASE.

Digital Library

[15]

Cuixiong Hu and Iulian Neamtiu. 2011. Automating GUI testing for Android applications. In Proceedings of the 6th International Workshop on Automation of Software Test. ACM, 77–83.

Digital Library

[16]

Timotej Kapus and Cristian Cadar. 2017. Automatic testing of symbolic execution engines via program generation and differential testing. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017. 590–600.

Digital Library

[17]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, Edinburgh, United Kingdom - June 09 - 11, 2014. 216–226.

Digital Library

[18]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2015). ACM, 386–399.

Digital Library

[19]

Daniel Lehmann and Michael Pradel. 2018. Feedback-directed Differential Testing of Interactive Debuggers. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). ACM, New York, NY, USA, 610–620.

Digital Library

[20]

Huai Liu, Fei-Ching Kuo, Dave Towey, and Tsong Yueh Chen. 2014. How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering 40, 1 (2014), 4–22.

Digital Library

[21]

Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 224–234.

Digital Library

[22]

William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100–107.

[23]

Michael Pradel and Koushik Sen. 2015. The Good, the Bad, and the Ugly: An Empirical Study of Implicit Type Conversions in JavaScript. In European Conference on Object-Oriented Programming (ECOOP).

[24]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, Beijing, China - June 11 - 16, 2012. 335–346.

Digital Library

[25]

Sergio Segura, Gordon Fraser, Ana B. Sánchez, and Antonio Ruiz Cortés. 2016. A Survey on Metamorphic Testing. IEEE Trans. Software Eng. 42, 9 (2016), 805–824.

[26]

Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique. In 17th Asia Pacific Software Engineering Conference, APSEC 2010, Sydney, Australia, November 30 - December 3, 2010. 270–279.

Digital Library

[27]

Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018. 303–314.

Digital Library

[28]

Xiaoyuan Xie, Joshua W. K. Ho, Christian Murphy, Gail E. Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 84, 4 (2011), 544–558.

Digital Library

[29]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In ACM SIGPLAN Notices, Vol. 46. ACM, 283–294.

Digital Library

[30]

Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal Program Enumeration for Rigorous Compiler Testing. In PLDI.

Digital Library

Cited By

Kitt LCohen M(2024)MorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum CompilersProceedings of the 2024 Workshop on Replications and Negative Results10.1145/3695750.3695823(15-21)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3695750.3695823
Kitt LCohen MFilkov VRay BZhou M(2024)MorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum CompilersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops10.1145/3691621.3694959(8-14)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691621.3694959
Zhang XLiu YArcaini PJiang MZheng Z(2024)MET-MAPF: A Metamorphic Testing Approach for Multi-Agent Path Finding AlgorithmsACM Transactions on Software Engineering and Methodology10.1145/366966333:8(1-37)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3669663
Show More Cited By

Index Terms

Interactive metamorphic testing of debuggers
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software maintenance tools

Recommendations

Feedback-directed differential testing of interactive debuggers
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

To understand, localize, and fix programming errors, developers often rely on interactive debuggers. However, as debuggers are software, they may themselves have bugs, which can make debugging unnecessarily hard or even cause developers to reason about ...
Automatic testing of interactive JavaScript debuggers
SPLASH Companion 2017: Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity

When debugging programs, we often assume the debugger itself is correct. However, when it is not, it becomes hard to find bugs or lets developers search for bugs that are not even present. We thus propose a new approach to automatic testing of ...
Self-Checked Metamorphic Testing of an Image Processing Program
SSIRI '10: Proceedings of the 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement

Metamorphic testing is an effective technique for testing systems that do not have test oracles, for which it is practically impossible to know the correct output of an arbitrary test input. In metamorphic testing, instead of checking the correctness of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2019

451 pages

ISBN:9781450362245

DOI:10.1145/3293882

General Chair:
Dongmei Zhang
Microsoft Research, China
,
Program Chair:
Anders Møller
Aarhus University, Denmark

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '19

Sponsor:

SIGSOFT

ISSTA '19: 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 15 - 19, 2019

Beijing, China

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
320
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)8

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kitt LCohen M(2024)MorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum CompilersProceedings of the 2024 Workshop on Replications and Negative Results10.1145/3695750.3695823(15-21)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3695750.3695823
Kitt LCohen MFilkov VRay BZhou M(2024)MorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum CompilersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops10.1145/3691621.3694959(8-14)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691621.3694959
Zhang XLiu YArcaini PJiang MZheng Z(2024)MET-MAPF: A Metamorphic Testing Approach for Multi-Agent Path Finding AlgorithmsACM Transactions on Software Engineering and Methodology10.1145/366966333:8(1-37)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3669663
Lu HLiu ZWang SZhang F(2024)DTD: Comprehensive and Scalable Testing for DebuggersProceedings of the ACM on Software Engineering10.1145/36437791:FSE(1172-1193)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643779
Méndez MBecerra-Terón AAlmendros-Jiménez JMerayo MNúñez M(2024)Combining Metamorphic Testing and Machine Learning to Enhance OpenStreetMapIEEE Transactions on Reliability10.1109/TR.2024.337936673:4(1834-1848)Online publication date: Dec-2024
https://doi.org/10.1109/TR.2024.3379366
Laurent TArcaini PZhang XIshikawa F(2024)Metamorphic Testing of an Autonomous Delivery Robots Scheduler2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00040(361-372)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00040
Yang YSun MWang YLi QWen MZhou YChandra SBlincoe KTonella P(2023)Heterogeneous Testing for Coverage Profilers Empowered with Debugging SupportProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616340(670-681)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616340
Zhang HPei YChen JTan SChandra SBlincoe KTonella P(2023)Statfier: Automated Testing of Static Analyzers via Semantic-Preserving Program TransformationsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616272(237-249)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616272
Wang YZhang PSun MLu ZYang YTang YQian JLi ZZhou Y(2023)Uncovering Bugs in Code Coverage Profilers via Control Flow Constraint SolvingIEEE Transactions on Software Engineering10.1109/TSE.2023.332138149:11(4964-4987)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3321381
Chen LZhou ZLi XJiang H(2023)Detecting JavaScript Transpiler Bugs with Grammar-guided Mutation2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00058(558-568)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00058
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents