research-article

Bayesian specification learning for finding API usage errors

Authors:

Vijayaraghavan Murali,

Swarat Chaudhuri,

Chris JermaineAuthors Info & Claims

ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

Pages 151 - 162

https://doi.org/10.1145/3106237.3106284

Published: 21 August 2017 Publication History

Abstract

We present a Bayesian framework for learning probabilistic specifications from large, unstructured code corpora, and then using these specifications to statically detect anomalous, hence likely buggy, program behavior. Our key insight is to build a statistical model that correlates all specifications hidden inside a corpus with the syntax and observed behavior of programs that implement these specifications. During the analysis of a particular program, this model is conditioned into a posterior distribution that prioritizes specifications that are relevant to the program. The problem of finding anomalies is now framed quantitatively, as a problem of computing a distance between a "reference distribution" over program behaviors that our model expects from the program, and the distribution over behaviors that the program actually produces.

We implement our ideas in a system, called Salento, for finding anomalous API usage in Android programs. Salento learns specifications using a combination of a topic model and a neural network model. Our encouraging experimental results show that the system can automatically discover subtle errors in Android applications in the wild, and has high precision and recall compared to competing probabilistic approaches.

References

[1]

Androiddrawer. http://www.androiddrawer.com. {Online; accessed 06-Jul-2016}.

[2]

F-Droid. https://f-droid.org. {Online; accessed 06-Jul-2016}.

[3]

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. ManÃľ, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. ViÃľgas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2015.

[4]

M. Allamanis and C. Sutton. Mining idioms from source code. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pages 472–483, New York, NY, USA, 2014. ACM.

Digital Library

[5]

N. S. Altman. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3):175–185, 1992.

[6]

R. Alur, P. Černý, P. Madhusudan, and W. Nam. Synthesis of interface specifications for java classes. In POPL, pages 98–109, 2005.

Digital Library

[7]

G. Ammons, R. Bodík, and J. R. Larus. Mining specifications. In POPL, pages 4–16, 2002.

Digital Library

[8]

G. Ammons, D. Mandelin, and J. R. Larus. Debugging temporal specifications with concept analysis. In In ACM SIGPLAN Conf on Prog Lang Design and Implem, pages 182–195. ACM Press, 2003.

Digital Library

[9]

S. Anand, C. S. Păsăreanu, and W. Visser. Jpf–se: A symbolic execution extension to java pathfinder. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 134–138. Springer, 2007.

Digital Library

[10]

G. K. Baah, A. Gray, and M. J. Harrold. On-line anomaly detection of deployed software: a statistical machine learning approach. In Proceedings of the 3rd international workshop on Software quality assurance, pages 70–77. ACM, 2006.

Digital Library

[11]

T. Ball, V. Levin, and S. K. Rajamani. A decade of software model checking with slam. Commun. ACM, 54(7):68–76, July 2011.

Digital Library

[12]

N. E. Beckman and A. V. Nori. Probabilistic, modular and scalable inference of typestate specifications. SIGPLAN Not., 46(6):211–221, 2011.

Digital Library

[13]

Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. journal of machine learning research, 3(Feb):1137–1155, 2003.

Digital Library

[14]

A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros, A. Kamsky, S. McPeak, and D. Engler. A few billion lines of code later: Using static analysis to find bugs in the real world. Communications of the ACM, 53(2):66–75, Feb. 2010.

Digital Library

[15]

P. Bielik, V. Raychev, and M. Vechev. Programming with "big code": Lessons, techniques and applications. In LIPIcs-Leibniz International Proceedings in Informatics, volume 32. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2015.

[16]

C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.

Digital Library

[17]

D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, 2003.

Digital Library

[18]

C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08, pages 209–224, Berkeley, CA, USA, 2008. USENIX Association.

Digital Library

[19]

T. M. Chilimbi and V. Ganapathy. Heapmd: Identifying heap-based bugs using anomaly detection. In ASPLOS, volume 34, pages 219–228. ACM, 2006.

Digital Library

[20]

D. R. Engler, D. Y. Chen, and A. Chou. Bugs as inconsistent behavior: A general approach to inferring errors in systems code. In SOSP, pages 57–72, 2001.

Digital Library

[21]

M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin. Dynamically discovering likely program invariants to support program evolution. In ICSE, pages 213–224, 1999.

Digital Library

[22]

Q. Fu, J.-G. Lou, Y. Wang, and J. Li. Execution anomaly detection in distributed systems through unstructured log analysis. In ICDM, volume 9, pages 149–158, 2009.

Digital Library

[23]

Q. Gao, F. Qin, and D. K. Panda. Dmtracker: finding bugs in large-scale parallel programs by detecting anomaly in data movements. In Proceedings of the 2007 ACM/IEEE conference on Supercomputing, page 15. ACM, 2007.

Digital Library

[24]

S. Geman and D. Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell., 6(6):721– 741, Nov. 1984.

Digital Library

[25]

C. Goues and W. Weimer. Specification mining with few false positives. In TACAS, pages 292–306, 2009.

Digital Library

[26]

X. Gu, H. Zhang, D. Zhang, and S. Kim. Deep API learning. CoRR, abs/1605.08535, 2016.

Digital Library

[27]

T. Gvero, V. Kuncak, I. Kuraj, and R. Piskac. Complete completion using types and weights. SIGPLAN Not., 48(6):27–38, June 2013.

Digital Library

[28]

S. Hangal and M. S. Lam. Tracking down software bugs using automatic anomaly detection. In Proceedings of the 24th international conference on Software engineering, pages 291–301. ACM, 2002.

Digital Library

[29]

J. Jaffar, V. Murali, J. A. Navas, and A. E. Santosa. Tracer: A symbolic execution tool for verification. In Proceedings of the 24th International Conference on Computer Aided Verification, CAV’12, pages 758–766, Berlin, Heidelberg, 2012. Springer-Verlag.

Digital Library

[30]

E. Jones, T. Oliphant, P. Peterson, et al. SciPy: Open source scientific tools for Python, 2001–. {Online; accessed 2016-11-09}.

[31]

J. C. King. Symbolic execution and program testing. Commun. ACM, 19(7):385– 394, July 1976.

Digital Library

[32]

T. Kremenek, A. Y. Ng, and D. Engler. A factor graph model for software bug finding. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, pages 2510–2516, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc.

Digital Library

[33]

T. Kremenek, P. Twohey, G. Back, A. Ng, and D. Engler. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, OSDI ’06, pages 161–176, Berkeley, CA, USA, 2006. USENIX Association.

Digital Library

[34]

S. Kullback and R. A. Leibler. On information and sufficiency. Ann. Math. Statist., 22(1):79–86, 03 1951.

[35]

Z. Li and Y. Zhou. Pr-miner: Automatically extracting implicit programming rules and detecting violations in large software code. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE-13, pages 306–315, New York, NY, USA, 2005. ACM.

Digital Library

[36]

B. Livshits, A. V. Nori, S. K. Rajamani, and A. Banerjee. Merlin: Specification inference for explicit information flow problems. In Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’09, pages 75–86, 2009.

Digital Library

[37]

T. Mikolov and G. Zweig. Context dependent recurrent neural network language model. In SLT, pages 234–239, 2012.

[38]

M. Monperrus, M. Bruch, and M. Mezini. Detecting missing method calls in object-oriented software. In Proceedings of the 24th European Conference on Object-oriented Programming, ECOOP’10, pages 2–25, Berlin, Heidelberg, 2010. Springer-Verlag.

Digital Library

[39]

A. S. Murawski and J. Ouaknine. Concur 2005 - concurrency theory. chapter On Probabilistic Program Equivalence and Refinement, pages 156–170. Springer-Verlag, London, UK, UK, 2005.

Digital Library

[40]

A. T. Nguyen and T. N. Nguyen. Graph-based statistical language model for code. In Proceedings of the 37th International Conference on Software Engineering - Volume 1, ICSE ’15, pages 858–868, Piscataway, NJ, USA, 2015. IEEE Press.

Digital Library

[41]

T. T. Nguyen, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen. A statistical semantic language model for source code. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 532–542, New York, NY, USA, 2013. ACM.

Digital Library

[42]

J. W. Nimmer and M. D. Ernst. Automatic generation of program specifications. In ISSTA 2002, Proceedings of the 2002 International Symposium on Software Testing and Analysis, pages 232–242, Rome, Italy, July 22–24, 2002.

Digital Library

[43]

D. Octeau, S. Jha, M. Dering, P. McDaniel, A. Bartel, L. Li, J. Klein, and Y. Le Traon. Combining static analysis with probabilistic models to enable market-scale android inter-component analysis. In Proceedings of the 43rd Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages, POPL 2016, pages 469–484, 2016.

Digital Library

[44]

V. Raychev, P. Bielik, and M. T. Vechev. Probabilistic model for code with decision trees. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016, Amsterdam, The Netherlands, October 30 - November 4, 2016, pages 731–747, 2016.

Digital Library

[45]

V. Raychev, P. Bielik, M. T. Vechev, and A. Krause. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016, pages 761–774, 2016.

Digital Library

[46]

V. Raychev, M. Vechev, and A. Krause. Predicting program properties from big code. In POPL, volume 50, pages 111–124, 2015.

Digital Library

[47]

V. Raychev, M. Vechev, and E. Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, pages 419–428, 2014.

Digital Library

[48]

S. Shoham, E. Yahav, S. Fink, and M. Pistoia. Static specification mining using automata-based abstractions. In Proceedings of the 2007 International Symposium on Software Testing and Analysis, ISSTA ’07, pages 174–184, New York, NY, USA, 2007. ACM.

Digital Library

[49]

A. Sokolova and E. P. de Vink. Probabilistic Automata: System Types, Parallel Composition and Comparison, pages 1–43. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004.

[50]

R. Vallée-Rai, P. Co, E. Gagnon, L. Hendren, P. Lam, and V. Sundaresan. Soot - a java bytecode optimization framework. In Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research, CASCON ’99, pages 13–. IBM Press, 1999.

Digital Library

[51]

J. Von Neumann. 13. various techniques used in connection with random digits. 1951.

[52]

S. Wang, D. Chollak, D. Movshovitz-Attias, and L. Tan. Bugram: Bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM International ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine Conference on Automated Software Engineering, ASE 2016, pages 708–719, New York, NY, USA, 2016. ACM.

Digital Library

[53]

S. Wang, D. Chollak, D. Movshovitz-Attias, and L. Tan. Bugram: bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016, pages 708–719, 2016.

Digital Library

[54]

A. Wasylkowski, A. Zeller, and C. Lindig. Detecting object usage anomalies. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering, ESEC-FSE ’07, pages 35–44, New York, NY, USA, 2007. ACM.

Digital Library

[55]

W. Weimer and G. C. Necula. Mining temporal specifications for error detection. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’05, pages 461–476, 2005.

Digital Library

[56]

J. Whaley, M. C. Martin, and M. S. Lam. Automatic extraction of object-oriented component interfaces. In Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’02, pages 218–228, New York, NY, USA, 2002. ACM.

Digital Library

[57]

H. Zhong, T. Xie, L. Zhang, J. Pei, and H. Mei. Mapo: Mining and recommending api usage patterns. In Proceedings of the 23rd European Conference on ECOOP 2009 — Object-Oriented Programming, Genoa, pages 318–343, 2009.

Digital Library

Cited By

Cai YYadavally AMishra AMontejo GNguyen TRoychoudhury APaiva AAbreu RStorey M(2024)Programming Assistant for Exception Handling with CodeBERTProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639188(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639188
Arteca ESchafer MTip F(2023)Learning How to Listen: Automatically Finding Bug Patterns in Event-Driven JavaScript APIsIEEE Transactions on Software Engineering10.1109/TSE.2022.314797549:1(166-184)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3147975
Khairunnesa SAhmed SImtiaz SRajan HLeavens G(2023)What kinds of contracts do ML APIs need?Empirical Software Engineering10.1007/s10664-023-10320-z28:6Online publication date: 17-Oct-2023
https://dl.acm.org/doi/10.1007/s10664-023-10320-z
Show More Cited By

Index Terms

Bayesian specification learning for finding API usage errors
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Anomaly detection
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
  2. Software organization and properties
    1. Software functional properties
      1. Formal methods
        Automated static analysis

Recommendations

App review driven collaborative bug finding
Abstract
Software development teams generally welcome any effort to expose bugs in their code base. In this work, we build on the hypothesis that mobile apps from the same category (e.g., two web browser apps) may be affected by similar bugs in their ...
Specification of generic APIs, or: why algebraic may be better than pre/post
HILT '14

Specification based on Floyd-Hoare logic, using pre- and post-conditions, is common in languages aimed at high integrity software. Such pre/postcondition specifications are geared towards verification of code. While this specification technique has ...
WYSIWIB: exploiting fine-grained program structure in a scriptable API-usage protocol-finding process

Bug-finding tools rely on specifications of what is correct or incorrect code. As it is difficult for a tool developer or user to anticipate all possible specifications, strategies for inferring specifications have been proposed. These strategies obtain ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

August 2017

1073 pages

ISBN:9781450351058

DOI:10.1145/3106237

General Chairs:
Eric Bodden
Paderborn University, Germany / Fraunhofer IEM, Germany
,
Wilhelm Schäfer
Paderborn University, Germany
,
Program Chairs:
Arie van Deursen
Delft University of Technology, Netherlands
,
Andrea Zisman
Open University, UK

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE'17

Sponsor:

SIGSOFT

ESEC/FSE'17: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering

September 4 - 8, 2017

Paderborn, Germany

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
700
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)4

Reflects downloads up to 23 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cai YYadavally AMishra AMontejo GNguyen TRoychoudhury APaiva AAbreu RStorey M(2024)Programming Assistant for Exception Handling with CodeBERTProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639188(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639188
Arteca ESchafer MTip F(2023)Learning How to Listen: Automatically Finding Bug Patterns in Event-Driven JavaScript APIsIEEE Transactions on Software Engineering10.1109/TSE.2022.314797549:1(166-184)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3147975
Khairunnesa SAhmed SImtiaz SRajan HLeavens G(2023)What kinds of contracts do ML APIs need?Empirical Software Engineering10.1007/s10664-023-10320-z28:6Online publication date: 17-Oct-2023
https://dl.acm.org/doi/10.1007/s10664-023-10320-z
Watson CCooper NPalacio DMoran KPoshyvanyk D(2022)A Systematic Literature Review on the Use of Deep Learning in Software Engineering ResearchACM Transactions on Software Engineering and Methodology10.1145/348527531:2(1-58)Online publication date: 4-Mar-2022
https://dl.acm.org/doi/10.1145/3485275
Schlichtig MSassalla SNarasimhan KBodden E(2022)FUM - A Framework for API Usage constraint and Misuse Classification2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00085(673-684)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00085
Li RChen BZhang FSun CPeng X(2022)Detecting Runtime Exceptions by Deep Code Representation Learning with Attention-Based Graph Neural Networks2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00053(373-384)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00053
Nielebock SBlockhaus PKruger JOrtmeier F(2021)An Experimental Analysis of Graph-Distance Algorithms for Comparing API Usages2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM52516.2021.00034(214-225)Online publication date: Sep-2021
https://doi.org/10.1109/SCAM52516.2021.00034
Zhang FChen BLi RPeng X(2021)A hybrid code representation learning approach for predicting method namesJournal of Systems and Software10.1016/j.jss.2021.111011180:COnline publication date: 1-Oct-2021
https://dl.acm.org/doi/10.1016/j.jss.2021.111011
Liu WChen BPeng XSun QZhao W(2021)Identifying change patterns of API misuses from code changesScience China Information Sciences10.1007/s11432-019-2745-564:3Online publication date: 7-Feb-2021
https://doi.org/10.1007/s11432-019-2745-5
Kashyap VScott RRanieri JMelski DKot LOh HSui Y(2020)API analytics for curating static analysis rulesProceedings of the 11th ACM SIGPLAN International Workshop on Tools for Automatic Program Analysis10.1145/3427764.3428318(1-2)Online publication date: 17-Nov-2020
https://dl.acm.org/doi/10.1145/3427764.3428318
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten