research-article

Amplifying Tests to Validate Exception Handling Code: An Extended Study in the Mobile Application Domain

Authors:

Sebastian ElbaumAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 23, Issue 4

Article No.: 32, Pages 1 - 28

https://doi.org/10.1145/2652483

Published: 05 September 2014 Publication History

Abstract

Validating code handling exceptional behavior is difficult, particularly when dealing with external resources that may be noisy and unreliable, as it requires (1) systematic exploration of the space of exceptions that may be thrown by the external resources, and (2) setup of the context to trigger specific patterns of exceptions. In this work, we first present a study quantifying the magnitude of the problem by inspecting the bug repositories of a set of popular applications in the increasingly relevant domain of Android mobile applications. The study revealed that 22% of the confirmed and fixed bugs have to do with poor exceptional handling code, and half of those correspond to interactions with external resources. We then present an approach that addresses this challenge by performing an systematic amplification of the program space explored by a test by manipulating the behavior of external resources. Each amplification attempts to expose a program's exception handling constructs to new behavior by mocking an external resource so that it returns normally or throws an exception following a predefined set of patterns. Our assessment of the approach indicates that it can be fully automated, is powerful enough to detect 67% of the faults reported in the bug reports of this kind, and is precise enough that 78% of the detected anomalies are fixed, and it has a great potential to assist developers.

References

[1]

Mithun Acharya and Tao Xie. 2009. Mining API error-handling specifications from source code. In Proceedings of the 12th International Conference on Fundamental Approaches to Software Engineering: held as part of the Joint European Conferences on Theory and Practice of Software (ETAPS/FASE'09). Springer-Verlag, Berlin, Heidelberg, 370--384.

Digital Library

[2]

Android-AspectJ Tool. 2011. http://code.google.com/p/android-aspectj/. (2011).

[3]

Android Developers Doc. 2011. http://developer.android.com/index.html. (2011).

[4]

Android Open Source DB. 2012. http://www.aopensource.com/. (2012).

[5]

Android Wifi Tether Code Host. 2012. http://code.google.com/p/myandroid-wifi-tether/. (2012).

[6]

AndroLib. 2012. http://www.androlib.com/appstats.aspx. (2012).

[7]

AppBrain. 2014. http://www.appbrain.com/stats/number-of-android-apps. (2014).

[8]

AspectJ Project. 2011. http://www.eclipse.org/aspectj/. (2011).

[9]

Eiji Adachi Barbosa, Alessandro Garcia, and Mira Mezini. 2012. Heuristic strategies for recommendation of exception handling code. In Proceedings of the 26th Brazilian Symposium on Software Engineering. 171--180.

Digital Library

[10]

Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010. A few billion lines of code later: Using static analysis to find bugs in the real world. Commun. ACM 53, 2 (2010), 66--75. http://doi.acm.org/10.1145/1646353.1646374

Digital Library

[11]

James M. Bieman, Daniel Dreilinger, and Lijun Lin. 1996. Using fault injection to increase software test coverage. In Proceedings of the International Symposium on Software Reliability Engineering. 166--174.

Digital Library

[12]

Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated testing based on Java predicates. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'02). ACM, New York, NY, 123--133.

Digital Library

[13]

Raymond P. L. Buse and Westley R. Weimer. 2008. Automatic documentation inference for exceptions. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'08). ACM, New York, NY, 273--282.

Digital Library

[14]

BusLinc Code Host. 2011. http://code.google.com/p/android-unl2011/. (2011).

[15]

Jong-Deok Choi, David Grove, Michael Hind, and Vivek Sarkar. 1999. Efficient and precise modeling of exceptions for the analysis of Java programs. SIGSOFT Softw. Eng. Notes 24, 5 (1999), 21--31.

Digital Library

[16]

Classification Study Dataset. 2012. http://cse.unl.edu/&sim;pzhang/data/tosem13.zip. (2012).

[17]

David Coppit, Jinlin Yang, Sarfraz Khurshid, Wei Le, and Kevin J. Sullivan. 2005. Software assurance by bounded exhaustive testing. IEEE Trans. Softw. Eng. 31, 4 (2005), 328--339.

Digital Library

[18]

Cyrket Mobile App Statistics. 2012. http://www.cyrket.com/m/android/. (2012).

[19]

Brett Daniel, Danny Dig, Kely Garcia, and Darko Marinov. 2007. Automated testing of refactoring engines. In Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC-FSE'07). ACM, New York, NY, 185--194.

Digital Library

[20]

EasyMock. 2012. http://easymock.org/. (2012).

[21]

Sebastian Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Jonathan Dokulil. 2006. Carving differential unit test cases from system test cases. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT'06/FSE-14). ACM, New York, NY, 253--264.

Digital Library

[22]

Steve Freeman and Nat Pryce. 2006. Evolving an embedded domain-specific language in Java. In Proceedings of the Companion to the 21st ACM SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA'06). ACM, New York, NY, 855--865.

Digital Library

[23]

Chen Fu and Barbara G. Ryder. 2007. Exception-chain analysis: Revealing exception handling architecture in Java server applications. In Proceedings of the International Conference on Software Engineering (ICSE'07). IEEE Computer Society, Washington, DC, 230--239.

Digital Library

[24]

Chen Fu, Barbara G. Ryder, Ana Milanova, and David Wonnacott. 2004. Testing of Java Web services for robustness. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and analysis (ISSTA'04). ACM, New York, NY, 23--34.

Digital Library

[25]

Daniel Jackson and Craig Damon. 1996. Elements of style: Analyzing a software design feature with a counterexample detector. IEEE Trans. Softw. Eng. 22, 7 (1996), 484--495.

Digital Library

[26]

JMock. 2011. http://www.jmock.org/index.html. (2011).

[27]

K-9 Mail Code Host. 2012. http://code.google.com/p/k9mail/. (2012).

[28]

Keepassdroid Code Host. 2011. http://code.google.com/p/keepassdroid/. (2011).

[29]

Le Wiki Koumbit. 2012. Open source Android applications. https://wiki.koumbit.net/AndroidFreeSoftware. (2012).

[30]

myTracks Code Host. 2011. http://code.google.com/p/mytracks/. (2011).

[31]

Open GPS Tracker Code Host. 2012. http://code.google.com/p/open-gpstracker/. (2012).

[32]

Open Street Map Wiki. 2012. http://wiki.openstreetmap.org/wiki/Android OpenSource. (2012).

[33]

M. P. Robillard, E. Bodden, D. Kawrykow, M. Mezini, and T. Ratchford. 2013. Automated API property inference techniques. IEEE Trans. Softw. Eng. 39, 5 (2013), 613--637.

Digital Library

[34]

Martin P. Robillard and Gail C. Murphy. 2003. Static analysis to support the evolution of exception structure in object-oriented systems. ACM Trans. Softw. Eng. Methodol. 12, 2 (2003), 191--221.

Digital Library

[35]

Michele Sama, Sebastian G. Elbaum, Franco Raimondi, David S. Rosenblum, and Zhimin Wang. 2010. Context-aware adaptive applications: Fault patterns and their automated identification. IEEE Trans. Softw. Eng. 36, 5 (2010), 644--661.

Digital Library

[36]

Z. Segall, D. Vrsalovic, D. Siewiorek, D. Yaskin, J. Kownacki, J. Barton, R. Dancey, A. Robinson, and T. Lin. 1988. FIAT-fault injection based automated testing environment. In Proceedings of the 18th International Symposium on Fault-Tolerant Computing. 102--107.

[37]

Saurabh Sinha and Mary Jean Harrold. 1999. Analysis and testing of programs with exception-handling constructs. IEEE Trans. Softw Eng. 26 (1999), 849--871.

Digital Library

[38]

Saurabh Sinha, Alessandro Orso, and Mary Jean Harrold. 2004. Automated Support for development, maintenance, and testing in the presence of implicit control flow. In Proceedings of the 26th International Conference on Software Engineering (ICSE'04). IEEE Computer Society, DC, 336--345.

Digital Library

[39]

SipDroidVoIP Code Host. 2011. http://code.google.com/p/sipdroid/. (2011).

[40]

Suresh Thummalapenta and Tao Xie. 2009. Mining exception-handling rules as sequence association rules. In Proceedings of the 31st International Conference on Software Engineering (ICSE'09). IEEE Computer Society, Washington, DC, 496--506.

Digital Library

[41]

Trac. 2012. Open source Android applications. http://trac.osuosl.org/trac/replicant/wiki/ListOfKnown FreeApps. (2012).

[42]

Wikipedia. 2012. List of open source Android application. http://en.wikipedia.org/wiki/List_of_open_source_Android_applications. (2012).

[43]

XBMC Remote Code Host. http://code.google.com/p/android-xbmcremote/. (2011).

[44]

Pingyu Zhang and Sebastian Elbaum. 2012. Amplifying tests to validate exception handling code. In Proceedings of the International Conference on Software Engineering (ICSE'12). IEEE Press, Piscataway, NJ, 595--605.

Digital Library

Cited By

da Silva AVieira RMesquita DGomes JRocha L(2024)Towards automatic labeling of exception handling bugs: A case study of 10 years bug-fixing in Apache HadoopEmpirical Software Engineering10.1007/s10664-024-10494-029:4Online publication date: 5-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10494-0
Pan MLu YPei YZhang TLi X(2023)Preference-wise Testing of Android Apps via Test AmplificationACM Transactions on Software Engineering and Methodology10.1145/351180432:1(1-37)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3511804
Zhang HLuo JHu MYan JZhang JQiu ZGrundy JPollock LPenta M(2023)Detecting Exception Handling Bugs in C++ ProgramsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00098(1084-1096)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00098
Show More Cited By

Index Terms

Amplifying Tests to Validate Exception Handling Code: An Extended Study in the Mobile Application Domain
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Amplifying tests to validate exception handling code
ICSE '12: Proceedings of the 34th International Conference on Software Engineering

Validating code handling exceptional behavior is difficult, particularly when dealing with external resources that may be noisy and unreliable, as it requires: 1) the systematic exploration of the space of exceptions that may be thrown by the external ...
Efficient Java exception handling in just-in-time compilation
Research Articles

Java uses exceptions to provide elegant error handling capabilities during program execution. However, the presence of exception handlers complicates the job of the just-in-time (JIT) compiler, while exceptions are rarely used in most programs. This ...
Code recommendation for exception handling
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Exception handling is an effective mechanism to avoid unexpected runtime errors. However, novice programmers might fail to handle exceptions properly, causing serious errors like system crashing or resource leaking. In this paper, we introduce ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 23, Issue 4

Special Issue International Conference on Software Engineering (ICSE 2012) and Regular Papers

August 2014

232 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2668018

Editors:
David S. Rosenblum
National University of Singapore, Singapore
,
Gail C. Murphy,
Mauro Pezzé

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 September 2014

Accepted: 01 June 2014

Revised: 01 May 2014

Received: 01 April 2013

Published in TOSEM Volume 23, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
625
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

da Silva AVieira RMesquita DGomes JRocha L(2024)Towards automatic labeling of exception handling bugs: A case study of 10 years bug-fixing in Apache HadoopEmpirical Software Engineering10.1007/s10664-024-10494-029:4Online publication date: 5-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10494-0
Pan MLu YPei YZhang TLi X(2023)Preference-wise Testing of Android Apps via Test AmplificationACM Transactions on Software Engineering and Methodology10.1145/351180432:1(1-37)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3511804
Zhang HLuo JHu MYan JZhang JQiu ZGrundy JPollock LPenta M(2023)Detecting Exception Handling Bugs in C++ ProgramsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00098(1084-1096)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00098
Khanna DSharma SPurandare R(2023)Verifying Exception-Handling Code in Concurrent Libraries2023 30th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC60848.2023.00051(405-414)Online publication date: 4-Dec-2023
https://doi.org/10.1109/APSEC60848.2023.00051
Fazzini MChoi CCopia JLee GKakehi YGorla AOrso ADwyer MDamian DZeller A(2022)Use of test doubles in Android testingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510175(2266-2278)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510175
Escobar-Velasquez CLinares-Vasquez MBavota GTufano MMoran KDi Penta MVendome CBernal-Cardenas CPoshyvanyk D(2022)Enabling Mutant Generation for Open- and Closed-Source Android AppsIEEE Transactions on Software Engineering10.1109/TSE.2020.298263848:1(186-208)Online publication date: 1-Jan-2022
https://doi.org/10.1109/TSE.2020.2982638
Inupakutika DRodriguez GAkopian DLama PChalela PRamirez A(2022)On the Performance of Cloud-Based mHealth Applications: A Methodology on Measuring Service Response Time and a Case StudyIEEE Access10.1109/ACCESS.2022.317485510(53208-53224)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3174855
Wimalasooriya CLicorish SCosta DMacDonell S(2022)A systematic mapping study addressing the reliability of mobile applicationsJournal of Systems and Software10.1016/j.jss.2021.111166186:COnline publication date: 9-May-2022
https://dl.acm.org/doi/10.1016/j.jss.2021.111166
Schoofs EAbdi MDemeyer S(2022)AmPyfier: Test amplification in PythonJournal of Software: Evolution and Process10.1002/smr.249034:11Online publication date: 24-Jul-2022
https://doi.org/10.1002/smr.2490
Zhang LMorin BHaller PBaudry BMonperrus M(2021)A Chaos Engineering System for Live Analysis and Falsification of Exception-Handling in the JVMIEEE Transactions on Software Engineering10.1109/TSE.2019.295487147:11(2534-2548)Online publication date: 1-Nov-2021
https://doi.org/10.1109/TSE.2019.2954871
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents