research-article

Open access

Automatically Reproducing Android Bug Reports using Natural Language Processing and Reinforcement Learning

Authors:

William G.J. HalfondAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 411 - 422

https://doi.org/10.1145/3597926.3598066

Published: 13 July 2023 Publication History

Abstract

As part of the process of resolving issues submitted by users via bug reports, Android developers attempt to reproduce and observe the crashes described by the bug reports. Due to the low-quality of bug reports and the complexity of modern apps, the reproduction process is non-trivial and time-consuming. Therefore, automatic approaches that can help reproduce Android bug reports are in great need. However, current approaches to help developers automatically reproduce bug reports are only able to handle limited forms of natural language text and struggle to successfully reproduce crashes for which the initial bug report had missing or imprecise steps. In this paper, we introduce a new fully automated approach to reproduce crashes from Android bug reports that addresses these limitations. Our approach accomplishes this by leveraging natural language processing techniques to more holistically and accurately analyze the natural language in Android bug reports and designing new techniques, based on reinforcement learning, to guide the search for successful reproducing steps. We conducted an empirical evaluation of our approach on 77 real world bug reports. Our approach achieved 67% precision and 77% recall in accurately extracting reproduction steps from bug reports, reproduced 74% of the total bug reports, and reproduced 64% of the bug reports that contained missing steps, significantly outperforming state of the art techniques.

References

[1]

2005. Wikipedia Page of Passive Voice. https://en.wikipedia.org/wiki/English_passive_voice#Identifying_the_English_passive

[2]

2017. Bug Report – Transistor 149. https://github.com/y20k/transistor/issues/149

[3]

2017. OpenIE 5.1 Github repository. https://github.com/dair-iitd/OpenIE-standalone

[4]

2018. Bug Report – Materialistic 1067. https://github.com/hidroh/materialistic/issues/1067

[5]

2018. Yakusu Official Website. https://sites.google.com/view/yakusumobile/home

[6]

2019. Bug Report – FDroidClient 1821. https://gitlab.com/fdroid/fdroidclient/issues/1821

[7]

2019. ReCDroid Github repository. https://github.com/AndroidTestBugReport/ReCDroid

[8]

2020. Bug Report – AnkiDroid 6432. https://github.com/ankidroid/Anki-Android/issues/6432

[9]

2023. Android Accessibility. https://support.google.com/accessibility/android/answer/7158690?hl=en

[10]

2023. Github Issue Tracker. https://github.com/issues

[11]

2023. Google Code Issue Tracker. https://code.google.com/archive/

[12]

2023. ReproBot Website. https://sites.google.com/usc.edu/reprobot/home

[13]

2023. UI Automator. https://developer.android.com/training/testing/ui-automator

[14]

David Adamo, Md Khorrom Khan, Sreedevi Koppula, and Renée Bryce. 2018. Reinforcement Learning for Android GUI Testing. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. ACM, Lake Buena Vista FL USA. 2–8. isbn:978-1-4503-6053-1 https://doi.org/10.1145/3278186.3278187

Digital Library

[15]

Nikita Bhutani, H V Jagadish, and Dragomir Radev. 2016. Nested Propositions in Open Information Extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas. 55–64. https://doi.org/10.18653/v1/D16-1006

[16]

Matthias Cetto, Christina Niklaus, André Freitas, and Siegfried Handschuh. 2018. Graphene: Semantically-Linked Propositions in Open Information Extraction. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA. 2300–2311.

[17]

Oscar Chaparro, Carlos Bernal-Cárdenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng. 2019. Assessing the Quality of the Steps to Reproduce in Bug Reports. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Tallinn Estonia. 86–96. isbn:978-1-4503-5572-8 https://doi.org/10.1145/3338906.3338947

Digital Library

[18]

Oscar Chaparro, Jing Lu, Fiorella Zampetti, Laura Moreno, Massimiliano Di Penta, Andrian Marcus, Gabriele Bavota, and Vincent Ng. 2017. Detecting Missing Information in Bug Descriptions. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, Paderborn Germany. 396–407. isbn:978-1-4503-5105-8 https://doi.org/10.1145/3106237.3106285

Digital Library

[19]

Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser Using Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar. 740–750. https://doi.org/10.3115/v1/D14-1082

[20]

Luciano Del Corro and Rainer Gemulla. 2013. ClausIE: Clause-Based Open Information Extraction. In Proceedings of the 22nd International Conference on World Wide Web - WWW ’13. ACM Press, Rio de Janeiro, Brazil. 355–366. isbn:978-1-4503-2035-1 https://doi.org/10.1145/2488388.2488420

Digital Library

[21]

Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S. Weld. 2008. Open Information Extraction from the Web. In Proceedings of the 20th International Joint Conference on Artifical Intelligence. 51, Morgan Kaufmann Publishers Inc., Hyderabad, India. 2670–2676. https://doi.org/10.5555/1625275.1625705

Digital Library

[22]

Mattia Fazzini, Kevin Patrick Moran, Carlos Bernal-Cardenas, Tyler Wendland, Alessandro Orso, and Denys Poshyvanyk. 2022. Enhancing Mobile App Bug Reporting via Real-time Understanding of Reproduction Steps. IEEE Transactions on Software Engineering, 1–1. issn:0098-5589, 1939-3520, 2326-3881 https://doi.org/10.1109/TSE.2022.3174028

Digital Library

[23]

Mattia Fazzini, Martin Prammer, Marcelo d’Amorim, and Alessandro Orso. 2018. Automatically Translating Bug Reports into Test Cases for Mobile Apps. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Amsterdam Netherlands. 141–152. isbn:978-1-4503-5699-2 https://doi.org/10.1145/3213846.3213869

Digital Library

[24]

Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. https://doi.org/10.5281/zenodo.1212303

[25]

Jack Johnson, Junayed Mahmud, Tyler Wendland, Kevin Moran, Julia Rubin, and Mattia Fazzini. 2022. An Empirical Investigation into the Reproduction of Bug Reports for Android Apps. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Honolulu, HI, USA. 321–322. isbn:978-1-66543-786-8 https://doi.org/10.1109/SANER53432.2022.00048

[26]

Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition (3rd ed.). 13, Prentice Hall PTR, USA. isbn:0130950696

Digital Library

[27]

Hui Liu, Mingzhu Shen, Jiahao Jin, and Yanjie Jiang. 2020. Automated Classification of Actions in Bug Reports of Mobile Apps. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual Event USA. 128–140. isbn:978-1-4503-8008-9 https://doi.org/10.1145/3395363.3397355

Digital Library

[28]

Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Comput. Linguist., 19, 2 (1993), jun, 313–330. issn:0891-2017

Digital Library

[29]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2. Curran Associates Inc., Red Hook, NY, USA. 9.

Digital Library

[30]

Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn Discourse Treebank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). European Language Resources Association (ELRA), 4.

[31]

Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, and Denys Poshyvanyk. 2015. Auto-Completing Bug Reports for Android Applications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, Bergamo Italy. 673–686. isbn:978-1-4503-3675-8 https://doi.org/10.1145/2786805.2786857

Digital Library

[32]

Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement Learning Based Curiosity-Driven Testing of Android Applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual Event USA. 153–164. isbn:978-1-4503-8008-9 https://doi.org/10.1145/3395363.3397354

Digital Library

[33]

Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), European Language Resources Association (ELRA), Marrakech, Morocco. 8.

[34]

Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open Language Learning for Information Extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, Jeju Island, Korea. 523–534.

[35]

Yang Song, Junayed Mahmud, Ying Zhou, Oscar Chaparro, Kevin Moran, Andrian Marcus, and Denys Poshyvanyk. 2022. Toward Interactive Bug Reporting for (Android App) End-Users. In In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’22). ACM, Singapore, Singapore. 13.

[36]

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA. isbn:0262039249

Digital Library

[37]

Thi Anh Tuyet Vuong and Shingo Takada. 2018. A Reinforcement Learning Based Approach to Automated Testing of Android Applications. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. ACM, Lake Buena Vista FL USA. 31–37. isbn:978-1-4503-6053-1 https://doi.org/10.1145/3278186.3278191

Digital Library

[38]

Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. In Machine learning. 279–292. https://aclanthology.org/P13-1045

[39]

Tyler Wendland, Jingyang Sun, Junayed Mahmud, S. M. Hasan Mansur, Steven Huang, Kevin Moran, Julia Rubin, and Mattia Fazzini. 2021. Andror2: A Dataset of Manually-Reproduced Bug Reports for Android Apps. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, Madrid, Spain. 600–604. isbn:978-1-72818-710-5 https://doi.org/10.1109/MSR52588.2021.00082

[40]

Fei Wu and Daniel S Weld. 2010. Open Information Extraction Using Wikipedia. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden. 118–127.

Digital Library

[41]

Yu Zhao, Kye Miller, Tingting Yu, Wei Zheng, and Minchao Pu. 2019. Automatically Extracting Bug Reproducing Steps from Android Bug Reports. In Reuse in the Big Data Era, Xin Peng, Apostolos Ampatzoglou, and Tanmay Bhowmik (Eds.). 11602, Springer International Publishing, Cham. 100–111. isbn:978-3-030-22887-3 978-3-030-22888-0 https://doi.org/10.1007/978-3-030-22888-0_8

[42]

Yu Zhao, Ting Su, Yang Liu, Wei Zheng, Xiaoxue Wu, Ramakanth Kavuluru, William G. J. Halfond, and Tingting Yu. 2022. ReCDroid+: Automated End-to-End Crash Reproduction from Bug Reports for Android Apps. ACM Transactions on Software Engineering and Methodology, 31, 3 (2022), July, 1–33. issn:1049-331X, 1557-7392 https://doi.org/10.1145/3488244

Digital Library

[43]

Yu Zhao, Tingting Yu, Ting Su, Yang Liu, Wei Zheng, Jingzhi Zhang, and William G.J. Halfond. 2019. ReCDroid: Automatically Reproducing Android Application Crashes from Bug Reports. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, Montreal, QC, Canada. 128–139. isbn:978-1-72810-869-8 https://doi.org/10.1109/ICSE.2019.00030

Digital Library

[44]

Yan Zheng, Yi Liu, Xiaofei Xie, Yepang Liu, Lei Ma, Jianye Hao, and Yang Liu. 2021. Automatic Web Testing Using Curiosity-Driven Reinforcement Learning. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, Madrid, ES. 423–435. isbn:978-1-66540-296-5 https://doi.org/10.1109/ICSE43902.2021.00048

Digital Library

[45]

Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What Makes a Good Bug Report? IEEE Transactions on Software Engineering, 36, 5 (2010), Sept., 618–643. issn:0098-5589 https://doi.org/10.1109/TSE.2010.63

Digital Library

Cited By

Wang DZhao YFeng SZhang ZHalfond WChen CSun XShi JYu TChristakis MPradel M(2024)Feedback-Driven Automated Whole Bug Report Reproduction for Android AppsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680341(1048-1060)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680341
Masoudian MHuang HAmini MZhang C(2024)Mole: Efficient Crash Reproduction in Android Applications With Enforcing Necessary UI EventsIEEE Transactions on Software Engineering10.1109/TSE.2024.342854350:8(2200-2218)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3428543
Cui ZLin GZheng LZhang Z(2024)A Fast Crash Reproduction Method for Android Applications Based on Widget Hierarchy GraphsIEEE Internet of Things Journal10.1109/JIOT.2024.335720911:8(13217-13230)Online publication date: 15-Apr-2024
https://doi.org/10.1109/JIOT.2024.3357209
Show More Cited By

Index Terms

Automatically Reproducing Android Bug Reports using Natural Language Processing and Reinforcement Learning
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

From Android Bug Reports to Android Bug Handling Process: An Empirical Study of Open-Source Development

Android is an operating system for mobile devices. Its development is led by Google and some other companies. Because of the open-source property of Android, anyone can report a bug through its online bug tracking system. In this paper, we analyze the ...
Generating Reproducible and Replayable Bug Reports from Android Application Crashes
ICPC '15: Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension

Manually reproducing bugs is time-consuming and tedious. Software maintainers routinely try to reproduce unconfirmed issues using incomplete or no informative bug reports. Consequently, while reproducing an issue, the maintainer must augment the report ...
Generating reproducible and replayable bug reports from Android application crashes
ICPC '15: Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension

Manually reproducing bugs is time-consuming and tedious. Software maintainers routinely try to reproduce unconfirmed issues using incomplete or noninformative bug reports. Consequently, while reproducing an issue, the maintainer must augment the report ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
516
Total Downloads

Downloads (Last 12 months)421
Downloads (Last 6 weeks)51

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang DZhao YFeng SZhang ZHalfond WChen CSun XShi JYu TChristakis MPradel M(2024)Feedback-Driven Automated Whole Bug Report Reproduction for Android AppsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680341(1048-1060)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680341
Masoudian MHuang HAmini MZhang C(2024)Mole: Efficient Crash Reproduction in Android Applications With Enforcing Necessary UI EventsIEEE Transactions on Software Engineering10.1109/TSE.2024.342854350:8(2200-2218)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3428543
Cui ZLin GZheng LZhang Z(2024)A Fast Crash Reproduction Method for Android Applications Based on Widget Hierarchy GraphsIEEE Internet of Things Journal10.1109/JIOT.2024.335720911:8(13217-13230)Online publication date: 15-Apr-2024
https://doi.org/10.1109/JIOT.2024.3357209
Aktas ECakmak EInan MYilmaz C(2024)Improving the quality of software issue report descriptions in Turkish: An industrial case study at SofttechEmpirical Software Engineering10.1007/s10664-023-10434-429:2Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1007/s10664-023-10434-4

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents