research-article

A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research

Authors:

David Nader Palacio,

Denys PoshyvanykAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 31, Issue 2

Article No.: 32, Pages 1 - 58

https://doi.org/10.1145/3485275

Published: 04 March 2022 Publication History

Abstract

An increasingly popular set of techniques adopted by software engineering (SE) researchers to automate development tasks are those rooted in the concept of Deep Learning (DL). The popularity of such techniques largely stems from their automated feature engineering capabilities, which aid in modeling software artifacts. However, due to the rapid pace at which DL techniques have been adopted, it is difficult to distill the current successes, failures, and opportunities of the current research landscape. In an effort to bring clarity to this cross-cutting area of work, from its modern inception to the present, this article presents a systematic literature review of research at the intersection of SE & DL. The review canvasses work appearing in the most prominent SE and DL conferences and journals and spans 128 papers across 23 unique SE tasks. We center our analysis around the components of learning, a set of principles that governs the application of machine learning techniques (ML) to a given problem domain, discussing several aspects of the surveyed work at a granular level. The end result of our analysis is a research roadmap that both delineates the foundations of DL techniques applied to SE research and highlights likely areas of fertile exploration for the future.

Supplementary Material

watson (watson.zip)

Supplemental movie, appendix, image and software files for, A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research

Download
37.91 KB

References

[1]

Alison Farr. 2016. Welcome to core. Welcome to Core. The Computing Research and Education Association of Australasia. https://www.core.edu.au/home.

[2]

Pornsiri Muenchaisri. Literature Reviews on Applying Artificial Intelligence/Machine Learning to Software Engineering Research Problems: Preliminary. SEED@APSEC.

[3]

Artifact Review and Badging – Version 2.0. Artifact Review and Badging – Version 2.0, Association for Computing Machinery, 4 Nov. 2021, https://www.acm.org/publications/policies/artifact-review-badging.

[4]

Mierswa, Ingo, and Ralf Klinkenberg. “Best Data Science & Machine Learning Platform.” RapidMiner, RapidMiner Inc., 9 Nov. 2021, https://rapidminer.com/.

[5]

Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin. 2012. Learning from Data: A Short Course. AMLbook.com.

[6]

Toufique Ahmed, V. Hellendoorn, and Premkumar T. Devanbu. 2021. Learning lenient parsing & typing via indirect supervision. Empir. Softw. Eng. 26 (2021), 1–31.

Digital Library

[7]

Feidu Akmel, Ermiyas Birhanu, and Bahir Siraj. 2018. A literature review study of software defect prediction using machine learning techniques. Int. J. Emerg. Res. Manag. Technol. 6 (06 2018), 300. DOI:

[8]

Miltiadis Allamanis. 2018. The adverse effects of code duplication in machine learning models of code. CoRR abs/1812.06469 (2018).

[9]

Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’15). ACM, New York, NY, 38–49. DOI:

Digital Library

[10]

Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=BJOFETxR-.

[11]

Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Proceedings of the 33rd International Conference on Machine Learning (JMLR Workshop and Conference Proceedings), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. JMLR.org, 2091–2100. Retrieved from http://proceedings.mlr.press/v48/allamanis16.html.

[12]

Plamen Angelov and Eduardo Soares. 2020. Towards explainable deep neural networks (xDNN). Neural Netw. 130 (2020), 185–194. DOI:

[13]

Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. 2018. Combining symbolic expressions and black-box function evaluations in neural programs. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=Hksj2WWAW.

[14]

Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to write programs. In Proceedings of the 5th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=ByldLrqlx.

[15]

Rohan Bavishi, Caroline Lemieux, Roy Fox, Koushik Sen, and Ion Stoica. 2019. AutoPandas: Neural-backed generators for program synthesis. Proc. ACM Program. Lang. 3, OOPSLA (Oct. 2019). DOI:

Digital Library

[16]

Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018. Neural code comprehension: A learnable representation of code semantics. In Proceedings of the Annual Conference on Neural Information Processing Systems, Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 3589–3601. Retrieved from https://proceedings.neurips.cc/paper/2018/hash/17c3433fecc21b57000debdf7ad5c930-Abstract.html.

[17]

Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh. 2018. Neuro-symbolic program corrector for introductory programming assignments. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, New York, NY, 60–70. DOI:

Digital Library

[18]

Nghi D. Q. Bui, Lingxiao Jiang, and Yijun Yu. 2018. Cross-language learning for program classification using bilateral tree-based convolutional neural networks. In Proceedings of the Workshops of the the 32nd AAAI Conference on Artificial Intelligence (AAAI Workshops’18). AAAI Press, 758–761. Retrieved from https://aaai.org/ocs/index.php/WS/AAAIW18/paper/view/17338.

[19]

Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. 2019. Bilateral dependency neural networks for cross-language algorithm classification. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 422–433. DOI:

[20]

Rudy Bunel, Matthew J. Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. 2018. Leveraging grammar and reinforcement learning for neural program synthesis. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=H1Xw62kRZ.

[21]

Lutz Büch and Artur Andrzejak. 2019. Learning-based recursive aggregation of abstract syntax trees for code clone detection. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 95–104. DOI:

[22]

Jonathon Cai, Richard Shin, and Dawn Song. 2017. Making neural programming architectures generalize via recursion. In Proceedings of the 5th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=BkbY4psgg.

[23]

Raghavendra Chalapathy and Sanjay Chawla. 2019. Deep learning for anomaly detection: A survey. CoRR abs/1901.03407 (2019).

[24]

K. Charmaz. 2006. Constructing Grounded Theory. SAGE Publications Inc.

[25]

Chao Chen, Wenrui Diao, Yingpei Zeng, Shanqing Guo, and Chengyu Hu. 2018. DRLgencert: Deep learning-based automated testing of certificate verification in SSL/TLS implementations. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution. IEEE Computer Society, 48–58. DOI:

[26]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI design image to GUI skeleton: A neural machine translator to bootstrap mobile GUI implementation. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, New York, NY, 665–676. DOI:

Digital Library

[27]

C. Chen, Z. Xing, Y. Liu, and K. L. X. Ong. 2019. Mining likely analogical APIs across third-party libraries via large-scale unsupervised API semantics embedding. IEEE Trans. Softw. Eng. 47, 3 (2019), 1–1. DOI:

[28]

G. Chen, C. Chen, Z. Xing, and B. Xu. 2016. Learning a dual-language vector space for domain-specific cross-lingual question retrieval. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 744–755.

Digital Library

[29]

Qingying Chen and Minghui Zhou. 2018. A neural framework for retrieval and summarization of source code. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 826–831. DOI:

Digital Library

[30]

Xinyun Chen, Chang Liu, and Dawn Song. 2018. Towards synthesizing complex programs from input-output examples. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=Skp1ESxRZ.

[31]

Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 2547–2557. Retrieved from http://papers.nips.cc/paper/7521-tree-to-tree-neural-networks-for-program-translation.pdf.

[32]

Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, and Aditya Ghose. 2017. Predicting the delay of issues with due dates in software projects. Empir. Softw. Eng. 22, 3 (01 June 2017), 1223–1263. DOI:

Digital Library

[33]

M. Choetkiertikul, H. K. Dam, T. Tran, A. Ghose, and J. Grundy. 2018. Predicting delivery capability in iterative software development. IEEE Trans. Softw. Eng. 44, 6 (June 2018), 551–573. DOI:

Digital Library

[34]

M. Choetkiertikul, H. K. Dam, T. Tran, T. Pham, A. Ghose, and T. Menzies. 2019. A deep learning model for estimating story points. IEEE Trans. Softw. Eng. 45, 7 (July 2019), 637–656. DOI:

[35]

C. S. Corley, K. Damevski, and N. A. Kraft. 2015. Exploring the use of deep learning for feature location. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’15). 556–560. DOI:

Digital Library

[36]

Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. 2018. Compiler fuzzing through deep learning. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’18). ACM, New York, NY, 95–105. DOI:

Digital Library

[37]

Milan Cvitkovic, Badal Singh, and Animashree Anandkumar. 2019. Open vocabulary learning on source code with a graph-structured cache. In Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, 1475–1485. Retrieved from http://proceedings.mlr.press/v97/cvitkovic19b.html.

[38]

Hoa Khanh Dam, Trang Pham, Shien Wee Ng, Truyen Tran, John Grundy, Aditya Ghose, Taeksu Kim, and Chul-Joo Kim. 2019. Lessons learned from using a deep tree-based model for software defect prediction in practice. In Proceedings of the IEEE/ACM 16th International Conference on Mining Software Repositories (MSR’19). 46–57. DOI:

Digital Library

[39]

H. K. Dam, T. Tran, T. T. M. Pham, S. W. Ng, J. Grundy, and A. Ghose. 2018. Automatic feature learning for predicting vulnerable software components. IEEE Trans. Softw. Eng. 47, 1 (2018), 1–1. DOI:

Digital Library

[40]

Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. 2017. Image-to-markup generation with coarse-to-fine attention. In Proceedings of the 34th International Conference on Machine Learning (ICML’17). JMLR.org, 980–989.

[41]

J. Deshmukh, A. K. M. S. Podder, S. Sengupta, and N. Dubash. 2017. Towards accurate duplicate bug retrieval using deep learning techniques. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’17). 115–124. DOI:

[42]

Jacob Devlin, Rudy Bunel, Rishabh Singh, Matthew J. Hausknecht, and Pushmeet Kohli. 2017. Neural program meta-induction. In Proceedings of the Annual Conference on Neural Information Processing Systems, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 2080–2088. Retrieved from https://proceedings.neurips.cc/paper/2017/hash/3bf55bbad370a8fcad1d09b005e278c2-Abstract.html.

[43]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018).

[44]

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural program learning under noisy I/O. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, 990–998. Retrieved from http://proceedings.mlr.press/v70/devlin17a.html.

[45]

Pedro Domingos. 1998. Occam’s two razors: The sharp and the blunt. In Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining (KDD’98). AAAI Press, 37–43.

[46]

John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12 (July 2011), 2121–2159. Retrieved from http://dl.acm.org/citation.cfm?id=1953048.2021068.

Digital Library

[47]

Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning libraries of subroutines for neurally guided Bayesian program induction. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 7805–7815. Retrieved from http://papers.nips.cc/paper/8006-learning-libraries-of-subroutines-for-neurallyguided-bayesian-program-induction.pdf.

[48]

Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning to infer graphics programs from hand-drawn images. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 6059–6068. Retrieved from http://papers.nips.cc/paper/7845-learning-to-infer-graphics-programs-from-hand-drawn-images.pdf.

[49]

Sarah Fakhoury, Venera Arnaoudova, Cedric Noiseux, Foutse Khomh, and Giuliano Antoniol. 2018. Keep it simple: Is deep learning good for linguistic smell detection? In Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER’18). 602–611. DOI:

[50]

Fabio Falcini, Giuseppe Lami, and Alessandra Costanza. 2017. Deep learning in automotive software. IEEE Softw. 34 (05 2017), 56–63. DOI:

Digital Library

[51]

F. Falcini, G. Lami, and A. Mitidieri. 2017. Yet another challenge for the automotive software: Deep learning. IEEE Softw. 34, 3 (2017), 1–1. DOI:

[52]

Ming Fan, Xiapu Luo, Jun Liu, Meng Wang, Chunyin Nong, Qinghua Zheng, and Ting Liu. 2019. Graph embedding based familial analysis of Android malware using unsupervised learning. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 771–782. DOI:

Digital Library

[53]

Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. From data mining to knowledge discovery in databases. AI Mag. 17, 3 (Mar. 1996), 37. DOI:

Digital Library

[54]

Fabio Ferreira, Luciana Lourdes Silva, and Marco Tulio Valente. 2019. Software Engineering Meets Deep Learning: A Literature Review. arxiv:1909.11436 [cs.SE]

[55]

Wei Fu and Tim Menzies. 2017. Easy over hard: A case study on deep learning. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). Association for Computing Machinery, New York, NY, 49–60. DOI:

Digital Library

[56]

Jian Gao, Xin Yang, Ying Fu, Yu Jiang, and Jiaguang Sun. 2018. VulSeeker: A semantic learning based vulnerability seeker for cross-platform binary. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 896–899. DOI:

Digital Library

[57]

Sa Gao, Chunyang Chen, Zhenchang Xing, Yukun Ma, Wen Song, and Shang-Wei Lin. 2019. A neural model for method name generation from functional description. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 414–421. DOI:

[58]

Alexander L. Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. 2017. Differentiable programs with neural libraries. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, 1213–1222. Retrieved from http://proceedings.mlr.press/v70/gaunt17a.html.

[59]

Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2019. Explaining Explanations: An Overview of Interpretability of Machine Learning. arxiv:1806.00069 [cs.AI]

[60]

Patrice Godefroid, Hila Peleg, and Rishabh Singh. 2017. Learn&Fuzz: Machine learning for input fuzzing. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE’17). IEEE Press, Piscataway, NJ, 50–59. Retrieved from http://dl.acm.org.proxy.wm.edu/citation.cfm?id=3155562.3155573.

Digital Library

[61]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. The MIT Press.

Digital Library

[62]

Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, New York, NY, 933–944. DOI:

Digital Library

[63]

Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API learning. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’16). ACM, New York, NY, 631–642. DOI:

Digital Library

[64]

Chenkai Guo, Dengrong Huang, Naipeng Dong, Quanqi Ye, Jing Xu, Yaqing Fan, Hui Yang, and Yifan Xu. 2019. Deep review sharing. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 61–72. DOI:

[65]

Jin Guo, Jinghui Cheng, and Jane Cleland-Huang. 2017. Semantically enhanced software traceability using deep learning techniques. In Proceedings of the 39th International Conference on Software Engineering (ICSE’17). IEEE Press, Piscataway, NJ, 3–14. DOI:

Digital Library

[66]

Jianmin Guo, Yu Jiang, Yue Zhao, Quan Chen, and Jiaguang Sun. 2018. DLFuzz: Differential fuzzing testing of deep learning systems. CoRR abs/1808.09413 (2018).

[67]

Rahul Gupta, Aditya Kanade, and Shirish Shevade. 2019. Deep reinforcement learning for syntactic error repair in student programs. Proc. AAAI Conf. Artif. Intell. 33 (07 2019), 930–937. DOI:

Digital Library

[68]

Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix: Fixing common C language errors by deep learning. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI’17). AAAI Press, 1345–1351.

[69]

Huong Ha and Hongyu Zhang. 2019. DeepPerf: Performance prediction for configurable software with deep sparse neural network. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 1095–1106. DOI:

Digital Library

[70]

Jiawei Han, Jian Pei, and Yiwen Yin. 2000. Mining frequent patterns without candidate generation. ACM SIGMOD Rec. 29, 2 (2000), 1–12.

[71]

Z. Han, X. Li, Z. Xing, H. Liu, and Z. Feng. 2017. Learning to predict severity of software vulnerability using only vulnerability description. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’17). 125–136. DOI:

[72]

Jacob Harer, Onur Ozdemir, Tomo Lazovich, Christopher Reale, Rebecca Russell, Louis Kim, and Peter Chin. 2018. Learning to repair software vulnerabilities with generative adversarial networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 7933–7943. Retrieved from http://papers.nips.cc/paper/8018-learning-to-repair-software-vulnerabilities-with-generative-adversarial-networks.pdf.

[73]

Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. 2018. Deep learning type inference. In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’18). ACM, New York, NY, 152–162. DOI:

Digital Library

[74]

Vincent J. Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code? In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). ACM, New York, NY, 763–773. DOI:

Digital Library

[75]

Vincent J. Hellendoorn, Premkumar T. Devanbu, and Mohammad Amin Alipour. 2018. On the naturalness of proofs. In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’18). ACM, New York, NY, 724–728. DOI:

Digital Library

[76]

Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. 2019. DeepJIT: An end-to-end deep learning framework for just-in-time defect prediction. In Proceedings of the IEEE/ACM 16th International Conference on Mining Software Repositories (MSR’19). 34–45. DOI:

Digital Library

[77]

Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension (ICPC’18). Association for Computing Machinery, New York, NY, 200–210. DOI:

Digital Library

[78]

Q. Huang, X. Xia, D. Lo, and G. C. Murphy. 2018. Automating intention mining. IEEE Trans. Softw. Eng. 46, 10 (2018), 1–1. DOI:

[79]

Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2019. Taxonomy of Real Faults in Deep Learning Systems. arxiv:1910.11015 [cs.SE]

[80]

Xuan Huo, Ferdian Thung, Ming Li, David Lo, and Shu-Ting Shi. 2021. Deep transfer bug localization. IEEE Trans. Softw. Eng. 47, 7 (2021), 1368–1380. DOI:

[81]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167 (2015).

[82]

Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically generating commit messages from diffs using neural machine translation. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE’17). IEEE Press, Piscataway, NJ, 135–146. Retrieved from http://dl.acm.org.proxy.wm.edu/citation.cfm?id=3155562.3155583.

[83]

Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. Neural-guided deductive search for real-time program synthesis from examples. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=rywDjg-RW.

[84]

Rafael-Michael Karampatsis and Charles Sutton. 2019. Maybe deep neural networks are the best choice for modeling source code. CoRR abs/1903.05734 (2019).

[85]

Rafael-Michael Karampatsis, Hlib Babii, R. Robbes, Charles Sutton, and A. Janes. 2020. Big code != Big vocabulary: Open-vocabulary models for source code. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering (ICSE’20). 1073–1085.

[86]

Deborah S. Katz, Jason Ruchti, and Eric Schulte. 2018. Using recurrent neural networks for decompilation. In Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER’18). 346–356. DOI:

[87]

Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. 2017. Learning How to Explain Neural Networks: PatternNet and PatternAttribution. arxiv:1705.05598 [stat.ML]

[88]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arxiv:1412.6980 [cs.LG]

[89]

B. Kitchenham and S. Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering (EBSE 2007-001). Keele University and Durham University Joint Report. Retrieved from https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf.

[90]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097–1105. Retrieved from http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf.

[91]

Donghwoon Kwon, H. Kim, Jinoh Kim, S. Suh, I. Kim, and K. J. Kim. 2017. A survey of deep learning-based network anomaly detection. Cluster Comput. 22 (2017), 949–961.

Digital Library

[92]

A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen. 2015. Combining deep learning with information retrieval to localize buggy files for bug reports (N). In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE’15). 476–481. DOI:

Digital Library

[93]

Tien-Duy B. Le, Lingfeng Bao, and David Lo. 2018. DSM: A specification mining tool using recurrent neural network based language model. In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’18). ACM, New York, NY, 896–899. DOI:

Digital Library

[94]

Tien-Duy B. Le and David Lo. 2018. Deep specification mining. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’18). Association for Computing Machinery, New York, NY, 106–117. DOI:

Digital Library

[95]

Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. A neural model for generating natural language summaries of program subroutines. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). IEEE Press, 795–806. DOI:

Digital Library

[96]

Sun-Ro Lee, Min-Jae Heo, Chan-Gun Lee, Milhan Kim, and Gaeul Jeong. 2017. Applying deep learning based automatic bug triager to industrial projects. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). ACM, New York, NY, 926–931. DOI:

Digital Library

[97]

Dor Levy and Lior Wolf. 2017. Learning to align the source code to the compiled object code. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, 2043–2051. Retrieved from http://proceedings.mlr.press/v70/levy17a.html.

[98]

D. Li, Z. Wang, and Y. Xue. 2018. Fine-grained Android malware detection based on deep learning. In Proceedings of the IEEE Conference on Communications and Network Security (CNS’18). 1–2. DOI:

[99]

Liuqing Li, He Feng, Wenjie Zhuang, Na Meng, and Barbara G. Ryder. 2017. CCLearner: A deep learning-based clone detection approach. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’17). 249–260.

[100]

Xiaochen Li, He Jiang, Zhilei Ren, Ge Li, and Jingxuan Zhang. 2018. Deep learning in software engineering. CoRR abs/1805.04825 (2018).

[101]

Yi Li, Shaohua Wang, Tien N. Nguyen, and Son Van Nguyen. 2019. Improving bug detection via context-based code representation learning and attention-based neural networks. Proc. ACM Program. Lang. 3, OOPSLA (Oct. 2019). DOI:

Digital Library

[102]

Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V. Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 9994–10006. Retrieved from http://papers.nips.cc/paper/8204-memory-augmented-policy-optimization-for-program-synthesis-and-semantic-parsing.pdf.

[103]

B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto. 2018. Sentiment analysis for software engineering: How far can we go? In Proceedings of the IEEE/ACM 40th International Conference on Software Engineering (ICSE’18). 94–104. DOI:

Digital Library

[104]

Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua Piao, and Wei Zou. 2018. \(\alpha\)Diff: Cross-version binary code similarity detection with DNN. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 667–678. DOI:

Digital Library

[105]

Chang Liu, Xinyun Chen, Richard Shin, Mingcheng Chen, and Dawn Song. 2016. Latent attention for if-then program synthesis. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 4574–4582. Retrieved from http://papers.nips.cc/paper/6284-latent-attention-for-if-then-program-synthesis.pdf.

[106]

Hui Liu, Zhifeng Xu, and Yanzhen Zou. 2018. Deep learning based feature envy detection. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 385–396. DOI:

Digital Library

[107]

K. Liu, D. Kim, T. F. Bissyande, S. Yoo, and Y. Le Traon. 2018. Mining fix patterns for FindBugs violations. IEEE Trans. Softw. Eng. 47, 1 (2018), 1–1. DOI:

Digital Library

[108]

Kui Liu, Dongsun Kim, Tegawendé F. Bissyandé, Taeyoung Kim, Kisub Kim, Anil Koyuncu, Suntae Kim, and Yves Le Traon. 2019. Learning to spot and refactor inconsistent method names. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 1–12.

Digital Library

[109]

P. Liu, X. Zhang, M. Pistoia, Y. Zheng, M. Marques, and L. Zeng. 2017. Automatic text input generation for mobile testing. In Proceedings of the IEEE/ACM 39th International Conference on Software Engineering (ICSE’17). 643–653. DOI:

Digital Library

[110]

Xiao Liu, Xiaoting Li, Rupesh Prajapati, and Dinghao Wu. 2019. DeepFuzz: Automatic generation of syntax valid C programs for fuzz testing. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference, and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Press, 1044–1051. DOI:

Digital Library

[111]

Yibin Liu, Yanhui Li, Jianbo Guo, Yuming Zhou, and Baowen Xu. 2018. Connecting software metrics across versions to predict defects. In Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER’18). 232–243. DOI:

[112]

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arxiv:2102.04664 [cs.SE]

[113]

Lei Ma, Felix Juefei-Xu, Jiyuan Sun, Chunyang Chen, Ting Su, Fuyuan Zhang, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Comprehensive and multi-granularity testing criteria for gauging the robustness of deep learning systems. CoRR abs/1803.07519 (2018).

[114]

Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 120–131. DOI:

Digital Library

[115]

Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and Ananth Grama. 2018. MODE: Automated neural network model debugging via state differential analysis and input selection. In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’18). ACM, New York, NY, 175–186. DOI:

Digital Library

[116]

David J. C. MacKay. 2002. Information Theory, Inference & Learning Algorithms. Cambridge University Press.

Digital Library

[117]

Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. 2019. NL2Type: Inferring JavaScript function types from natural language information. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 304–315. DOI:

Digital Library

[118]

Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable abstract interpretation for provably robust neural networks. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, 3578–3586. Retrieved from http://proceedings.mlr.press/v80/mirman18b.html.

[119]

Facundo Molina, Renzo Degiovanni, Pablo Ponzio, Germán Regis, Nazareno Aguirre, and Marcelo Frias. 2019. Training binary classifiers as data structure invariants. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). IEEE Press, 759–770. DOI:

Digital Library

[120]

K. P. Moran, C. Bernal-Cárdenas, M. Curcio, R. Bonett, and D. Poshyvanyk. 2018. Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans. Softw. Eng. 46, 2 (2018), 1–1. DOI:

[121]

Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI’16). AAAI Press, 1287–1293.

[122]

Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. 2017. Bayesian specification learning for finding API usage errors. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). ACM, New York, NY, 151–162. DOI:

Digital Library

[123]

Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. 2018. Neural sketch learning for conditional program generation. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=HkfXMz-Ab.

[124]

Anh Tuan Nguyen, Trong Duc Nguyen, Hung Dang Phan, and Tien N. Nguyen. 2018. A deep neural network language model with contexts for source code. In Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER’18). 323–334. DOI:

[125]

Sunday Oke. 2008. A literature review on artificial intelligence. Int. J. Inf. Manag. Sci. 19 (12 2008), 535–570.

[126]

J. Ott, A. Atchison, P. Harnack, A. Bergh, and E. Linstead. 2018. A deep learning approach to identifying source code in images and video. In Proceedings of the IEEE/ACM 15th International Conference on Mining Software Repositories (MSR’18). 376–386.

Digital Library

[127]

Stefano Panzeri, Cesare Magri, and Ludovico Carraro. 2008. Sampling bias. Scholarpedia, 3(9):4258. Retrieved from http://www.scholarpedia.org/article/Sampling_bias.

[128]

Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017. Neuro-symbolic program synthesis. In Proceedings of the 5th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=rJ0JwFcex.

[129]

Daniel Perez and Shigeru Chiba. 2019. Cross-language clone detection by learning over abstract syntax trees. In Proceedings of the IEEE/ACM 16th International Conference on Mining Software Repositories (MSR’19). 518–528. DOI:

Digital Library

[130]

Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. 2015. Learning program embeddings to propagate feedback on student code. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research), Francis Bach and David Blei (Eds.), Vol. 37. PMLR, 1093–1102. Retrieved from http://proceedings.mlr.press/v37/piech15.html.

[131]

Hans Plesser. 2018. Reproducibility vs. replicability: A brief history of a confused terminology. Front. Neuroinf. 11 (01 2018). DOI:

[132]

M. Pradel, Georgios Gousios, J. Liu, and S. Chandra. 2020. TypeWriter: Neural type prediction with search-based validation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.

[133]

Michael Pradel and Koushik Sen. 2018. DeepBugs: A learning approach to name-based bug detection. Proc. ACM Program. Lang. 2, OOPSLA (Oct. 2018). DOI:

Digital Library

[134]

Devanbu Prem, Matthew Dwyer, Sebastian Elbaum, Michael Lowry, Kevin Moran, Denys Poshyvanyk, Baishakhi Ray, Rishabh Singh, and Xiangyu Zhang. 2019. Deep learning & software engineering: State of research and future directions. In Proceedings of the NSF Workshop on Deep Learning and Software Engineering.

[135]

Carl Edward Rasmussen and Zoubin Ghahramani. 2001. Occam’s razor. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, T. K. Leen, T. G. Dietterich, and V. Tresp (Eds.). The MIT Press, 294–300. Retrieved from http://papers.nips.cc/paper/1925-occams-razor.pdf.

[136]

Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’14). Association for Computing Machinery, New York, NY, 419–428. DOI:

Digital Library

[137]

Scott E. Reed and Nando de Freitas. 2016. Neural programmer-interpreters. In Proceedings of the 4th International Conference on Learning Representations, Yoshua Bengio and Yann LeCun (Eds.). Retrieved from http://arxiv.org/abs/1511.06279.

[138]

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 4902–4912. DOI:

[139]

S. Romansky, N. C. Borle, S. Chowdhury, A. Hindle, and R. Greiner. 2017. Deep green: Modelling time-series of software energy consumption. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’17). 273–283. DOI:

[140]

Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu, Pierre Baldi, and Cristina V. Lopes. 2018. Oreo: Detection of clones in the twilight zone. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 354–365. DOI:

Digital Library

[141]

Neil Salkind. 2010. Encyclopedia of research design. CoRR 10.4135/9781412961288 (2010).

[142]

J. Sayyad Shirabad and T. J. Menzies. 2005. The PROMISE repository of software engineering databases.School of Information Technology and Engineering, University of Ottawa, Canada. Retrieved from http://promise.site.uottawa.ca/SERepository.

[143]

Jan Schroeder, Christian Berger, Alessia Knauss, Harri Preenja, Mohammad Ali, Miroslaw Staron, and Thomas Herpel. 2017. Predicting and evaluating software model growth in the automotive industry. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution. IEEE Computer Society, 584–593. DOI:

[144]

Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. 2019. Automatic code review by learning the revision of source code. Proc. AAAI Conf. Artif. Intell. 33 (07 2019), 4910–4917. DOI:

Digital Library

[145]

Richard Shin, Illia Polosukhin, and Dawn Song. 2018. Improving neural program synthesis with inferred execution traces. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 8917–8926. Retrieved from http://papers.nips.cc/paper/8107-improving-neural-program-synthesis-with-inferred-execution-traces.pdf.

[146]

Xujie Si, Hanjun Dai, Mukund Raghothaman, Mayur Naik, and Le Song. 2018. Learning loop invariants for program verification. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 7751–7762. Retrieved from http://papers.nips.cc/paper/8001-learning-loop-invariants-for-program-verification.pdf.

[147]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 56 (2014), 1929–1958. Retrieved from http://jmlr.org/papers/v15/srivastava14a.html.

Digital Library

[148]

Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: A critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering (ICSE’16). ACM, New York, NY, 120–131. DOI:

Digital Library

[149]

Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. 2018. Neural program synthesis from diverse demonstration videos. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, 4790–4799. Retrieved from http://proceedings.mlr.press/v80/sun18a.html.

[150]

Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. 2018. Concolic testing for deep neural networks. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 109–119. DOI:

Digital Library

[151]

Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang. 2019. A grammar-based structural CNN decoder for code generation. Proc. AAAI Conf. Artif. Intell. 33 (07 2019), 7055–7062. DOI:

Digital Library

[152]

J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia. 2014. Towards a big data curated benchmark of inter-project code clones. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution. 476–480.

Digital Library

[153]

Daniel Tarlow, Subhodeep Moitra, A. Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and E. Aftandilian. 2020. Learning to fix build errors with Graph2Diff neural networks. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops.

[154]

I. Tetko, D. Livingstone, and A. I. Luik. 1995. Neural network studies, 1. Comparison of overfitting and overtraining. J. Chem. Inf. Comput. Sci. 35 (1995), 826–833.

[155]

Hannes Thaller, Lukas Linsbauer, and Alexander Egyed. 2019. Feature maps: A comprehensible software representation for design pattern detection. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 207–217. DOI:

[156]

Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, New York, NY, 303–314. DOI:

Digital Library

[157]

Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk. 2019. On learning meaningful code changes via neural machine translation. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). IEEE Press, 25–36. DOI:

Digital Library

[158]

Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2018. Deep learning similarities from different representations of source code. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR’18). ACM, New York, NY, 542–553. DOI:

Digital Library

[159]

Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2018. An empirical investigation into learning bug-fixing patches in the wild via neural machine translation. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 832–837. DOI:

Digital Library

[160]

Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2018. An empirical study on learning bug-fixing patches in the wild via neural machine translation. CoRR abs/1812.08693 (2018).

[161]

Anne-Wil Harzing “Publish or Perish.”Publish or Perish, 5 Sept. 2019, https://harzing.com/resources/publish-or-perish.

[162]

Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 397–407. DOI:

Digital Library

[163]

Ke Wang, Rishabh Singh, and Zhendong Su. 2018. Dynamic neural program embeddings for program repair. In Proceedings of the 6th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=BJuWrGW0Z.

[164]

Song Wang, Taiyue Liu, Jaechang Nam, and Lin Tan. 2020. Deep semantic feature learning for software defect prediction. IEEE Trans. Softw. Eng. 46, 12 (2020), 1267–1293. DOI:

[165]

S. Wang, T. Liu, and L. Tan. 2016. Automatically learning semantic features for defect prediction. In Proceedings of the IEEE/ACM 38th International Conference on Software Engineering (ICSE’16). 297–308. DOI:

Digital Library

[166]

Shaohua Wang, NhatHai Phan, Yan Wang, and Yong Zhao. 2019. Extracting API tips from developer question and answer websites. In Proceedings of the IEEE/ACM 16th International Conference on Mining Software Repositories (MSR’19). 321–332. DOI:

Digital Library

[167]

Cody Watson, Nathan Cooper, David Nader, Kevin Moran, and Denys Poshyvanyk. 2021. Data Analysis for the Systematic Literature Review of DL4SE. Zenodo. DOI:

[168]

Cody Watson, David Palacio, Nathan Cooper, Kevin Moran, and Denys Poshyvanyk. 2021. Data analysis for the systematic literature review of DL4SE. Retrieved from https://wm-semeru.github.io/dl4se/.

[169]

Jianfeng Wen, Shixian Li, Zhiyong Lin, Yong Hu, and Changqin Huang. 2012. Systematic literature review of machine learning based software development effort estimation models. Inf. Softw. Technol. 54, 1 (2012), 41–59. DOI:

Digital Library

[170]

M. Wen, R. Wu, and S. C. Cheung. 2018. How well do change sequences predict defects? Sequence learning from software changes. IEEE Trans. Softw. Eng. 46, 11 (2018), 1–1. DOI:

[171]

Martin White, Michele Tufano, Matías Martínez, Martin Monperrus, and Denys Poshyvanyk. 2019. Sorting and transforming program repair ingredients via deep learning code similarities. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 479–490. DOI:

[172]

M. White, M. Tufano, C. Vendome, and D. Poshyvanyk. 2016. Deep learning code fragments for code clone detection. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 87–98.

Digital Library

[173]

Martin White, Christopher Vendome, Mario Linares-Vásquez, and Denys Poshyvanyk. 2015. Toward deep learning software repositories. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR’15). IEEE Press, Piscataway, NJ, 334–345. Retrieved from http://dl.acm.org/citation.cfm?id=2820518.2820559.

Digital Library

[174]

Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran. 2020. Explainable deep learning: A field guide for the uninitiated. arxiv:2004.14545 [cs.LG]

[175]

Rui Xie, Long Chen, Wei Ye, Zhiyu Li, Tianxiang Hu, Dongdong Du, and Shikun Zhang. 2019. DeepLink: A code knowledge graph based deep learning approach for issue-commit link recovery. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 434–444. DOI:

[176]

B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li. 2016. Predicting semantically linkable knowledge in developer online forums via convolutional neural network. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 51–62.

Digital Library

[177]

X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun. 2015. Deep learning for just-in-time defect prediction. In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security. 17–26.

Digital Library

[178]

Zebin Yang, Aijun Zhang, and Agus Sudjianto. 2019. Enhancing explainability of neural networks through architecture constraints. arxiv:1901.03838 [stat.ML]

[179]

Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR’18). Association for Computing Machinery, New York, NY, 476–486. DOI:

Digital Library

[180]

Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L. Gaunt. 2019. Learning to represent edits. In Proceedings of the 7th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=BJl6AjC5F7.

[181]

Hao Yu, Wing Lam, Long Chen, Ge Li, Tao Xie, and Qianxiang Wang. 2019. Neural detection of semantic code clones via tree-based convolution. In Proceedings of the 27th International Conference on Program Comprehension (ICPC’19). IEEE Press, 70–80. DOI:

Digital Library

[182]

Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. 2021. Explainability in graph neural networks: A taxonomic survey. arxiv:2012.15445 [cs.LG]

[183]

Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding DQNs. CoRR abs/1602.02658 (2016).

[184]

Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arxiv:1212.5701 [cs.LG]

[185]

Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 783–794. DOI:

Digital Library

[186]

Lisa Zhang, Gregory Rosenblatt, Ethan Fetaya, Renjie Liao, William Byrd, Matthew Might, Raquel Urtasun, and Richard Zemel. 2018. Neural guided constraint logic programming for program synthesis. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 1737–1746. Retrieved from http://papers.nips.cc/paper/7445-neural-guided-constraint-logic-programming-for-program-synthesis.pdf.

[187]

Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). ACM, New York, NY, 132–142. DOI:

Digital Library

[188]

Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’18). ACM, New York, NY, 129–140. DOI:

Digital Library

[189]

Zhuo Zhang, Yan Lei, Xiaoguang Mao, and Panpan Li. 2019. CNN-FL: An effective approach for localizing faults using convolutional neural networks. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). 445–455. DOI:

[190]

Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, and Guoqiang Li. 2019. ActionNet: Vision-based workflow action recognition from programming screencasts. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 350–361. DOI:

Digital Library

[191]

Gang Zhao and Jeff Huang. 2018. DeepSim: Deep learning code functional similarity. In Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’18). Association for Computing Machinery, New York, NY, 141–151. DOI:

Digital Library

[192]

Hui Zhao, Zhihui Li, Hansheng Wei, Jianqi Shi, and Yanhong Huang. 2019. SeqFuzzer: An industrial protocol fuzzing framework from a deep learning perspective. In Proceedings of the 12th IEEE Conference on Software Testing, Validation and Verification (ICST’19). 59–67. DOI:

[193]

Jinman Zhao, Aws Albarghouthi, Vaibhav Rastogi, Somesh Jha, and Damien Octeau. 2018. Neural-augmented static analysis of Android communication. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 342–353. DOI:

Digital Library

[194]

Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. CoRR abs/1702.04595 (2017).

[195]

Amit Zohar and Lior Wolf. 2018. Automatic program synthesis of long programs with a learned garbage collector. In Proceedings of the International Conference on Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 2094–2103. Retrieved from http://papers.nips.cc/paper/7479-automatic-program-synthesis-of-long-programs-with-a-learned-garbage-collector.pdf.

[196]

Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.

Cited By

Di Rocco JNguyen PDi Sipio CRubei RDi Ruscio DDi Penta M(2025)DeepMigInformation and Software Technology10.1016/j.infsof.2024.107588177:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.infsof.2024.107588
Zhao YGong LYu YHuang ZWei M(2025)An empirical study of best practices for code pre-trained models on software engineering classification tasksExpert Systems with Applications10.1016/j.eswa.2025.126762272(126762)Online publication date: May-2025
https://doi.org/10.1016/j.eswa.2025.126762
Zheng ZNing KZhong QChen JChen WGuo LWang WWang Y(2025)Towards an understanding of large language models in software engineering tasksEmpirical Software Engineering10.1007/s10664-024-10602-030:2Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1007/s10664-024-10602-0
Show More Cited By

Index Terms

A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques

Recommendations

Machine Learning and Software Engineering
ICTAI '02: Proceedings of the 14th IEEE International Conference on Tools with Artificial Intelligence

Machine learning deals with the issue of how to build programs that improve their performance at some task through experience. Machine learning algorithms have proven to be of great practical value in a variety of application domains. Not surprisingly, ...
Automating Software Engineering with Machine Learning
ISEC '22: Proceedings of the 15th Innovations in Software Engineering Conference

Software plays a crucial role in our everyday lives. The scarcity of skilled software engineers has become a bottleneck in delivering better software at scale. Can we automate software engineering to help improve developer productivity and software ...
RETRACTED ARTICLE: A Novel Framework in Software Engineering for Deep Learning
Abstract
There are increasingly deep learning approaches being used in software engineering domain. While, there are no agreement on how to adopt deep learning in this new domain. This work presents a methodological framework for deep learning in software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 31, Issue 2

April 2022

789 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3492439

Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 March 2022

Accepted: 01 September 2021

Revised: 01 July 2021

Received: 01 September 2020

Published in TOSEM Volume 31, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

NSF

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
5,598
Total Downloads

Downloads (Last 12 months)1,558
Downloads (Last 6 weeks)99

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Di Rocco JNguyen PDi Sipio CRubei RDi Ruscio DDi Penta M(2025)DeepMigInformation and Software Technology10.1016/j.infsof.2024.107588177:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.infsof.2024.107588
Zhao YGong LYu YHuang ZWei M(2025)An empirical study of best practices for code pre-trained models on software engineering classification tasksExpert Systems with Applications10.1016/j.eswa.2025.126762272(126762)Online publication date: May-2025
https://doi.org/10.1016/j.eswa.2025.126762
Zheng ZNing KZhong QChen JChen WGuo LWang WWang Y(2025)Towards an understanding of large language models in software engineering tasksEmpirical Software Engineering10.1007/s10664-024-10602-030:2Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1007/s10664-024-10602-0
Elgendy IHierons RMcMinn P(2025)A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing TechniquesSoftware Testing, Verification and Reliability10.1002/stvr.191435:2Online publication date: 17-Jan-2025
https://doi.org/10.1002/stvr.1914
Gong JChen T(2024)Deep Configuration Performance Learning: A Systematic Survey and TaxonomyACM Transactions on Software Engineering and Methodology10.1145/370298634:1(1-62)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3702986
Hou XZhao YLiu YYang ZWang KLi LLuo XLo DGrundy JWang H(2024)Large Language Models for Software Engineering: A Systematic Literature ReviewACM Transactions on Software Engineering and Methodology10.1145/369598833:8(1-79)Online publication date: 20-Sep-2024
https://dl.acm.org/doi/10.1145/3695988
Zhao YGong LHuang ZWang YWei MWu FFilkov VRay BZhou M(2024)Coding-PTMs: How to Find Optimal Code Pre-trained Models for Code Embedding in Vulnerability Detection?Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695539(1732-1744)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695539
Chen JZhang YZhang LZhang MWan CSu TPu GFilkov VRay BZhou M(2024)FIPSER: Improving Fairness Testing of DNN by Seed PrioritizationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695486(1069-1081)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695486
Dong LLu QZhu LMenzies TXu BKang HZhang J(2024)A Pilot Study in Surveying Data Challenges of Automatic Software Engineering TasksProceedings of the 4th International Workshop on Software Engineering and AI for Data Quality in Cyber-Physical Systems/Internet of Things10.1145/3663530.3665020(6-11)Online publication date: 15-Jul-2024
https://dl.acm.org/doi/10.1145/3663530.3665020
Yang HNong YZhang TLuo XCai H(2024)Learning to Detect and Localize Multilingual BugsProceedings of the ACM on Software Engineering10.1145/36608041:FSE(2190-2213)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660804
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents