research-article

Bugram: bug detection with n-gram language models

Authors:

Dana Movshovitz-Attias,

Lin TanAuthors Info & Claims

ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Pages 708 - 719

https://doi.org/10.1145/2970276.2970341

Published: 25 August 2016 Publication History

Abstract

To improve software reliability, many rule-based techniques have been proposed to infer programming rules and detect violations of these rules as bugs. These rule-based approaches often rely on the highly frequent appearances of certain patterns in a project to infer rules. It is known that if a pattern does not appear frequently enough, rules are not learned, thus missing many bugs.

In this paper, we propose a new approach—Bugram—that leverages n-gram language models instead of rules to detect bugs. Bugram models program tokens sequentially, using the n-gram language model. Token sequences from the program are then assessed according to their probability in the learned model, and low probability sequences are marked as potential bugs. The assumption is that low probability token sequences in a program are unusual, which may indicate bugs, bad practices, or unusual/special uses of code of which developers may want to be aware.

We evaluate Bugram in two ways. First, we apply Bugram on the latest versions of 16 open source Java projects. Results show that Bugram detects 59 bugs, 42 of which are manually verified as correct, 25 of which are true bugs and 17 are code snippets that should be refactored. Among the 25 true bugs, 23 cannot be detected by PR-Miner. We have reported these bugs to developers, 7 of which have already been confirmed by developers (4 of them have already been fixed), while the rest await confirmation. Second, we further compare Bugram with three additional graph- and rule-based bug detection tools, i.e., JADET, Tikanga, and GrouMiner. We apply Bugram on 14 Java projects evaluated in these three studies. Bugram detects 21 true bugs, at least 10 of which cannot be detected by these three tools. Our results suggest that Bugram is complementary to existing rule-based bug detection approaches.

References

[1]

M. Acharya, T. Xie, J. Pei, and J. Xu. Mining API Patterns as Partial Orders from Source Code: From Usage Scenarios to Specifications. In FSE’ 07, pages 25–34, 2007.

Digital Library

[2]

M. Allamanis, E. T. Barr, and C. Sutton. Learning Natural Coding Conventions. In FSE’ 14, pages 281–293, 2014.

Digital Library

[3]

L. R. Bahl, P. Brown, P. V. de Souza, and R. Mercer. A Tree-Based Statistical Language Model for Natural Language Speech Recognition. Acoustics, Speech and Signal Processing, IEEE Transactions on, 37(7):1001–1008, 1989.

[4]

L. Benjamin and T. Zimmermann. DynaMine: Finding Common Error Patterns by Mining Software Revision Histories. In FSE’ 05, pages 296–305, 2005.

[5]

J. C. Campbell, A. Hindle, and J. N. Amaral. Syntax Errors Just Aren’t Natural: Improving Error Reporting with Language Models. In MSR’ 14, pages 252–261, 2014.

Digital Library

[6]

R.-Y. Chang, A. Podgurski, and J. Yang. Finding What’s Not There: A New Approach to Revealing Neglected Conditions in Software. In ISSTA’ 07, pages 163–173, 2007.

Digital Library

[7]

E. Charniak. Statistical Language Learning. In First MIT Press paperback edition. MIT Press, 1996.

Digital Library

[8]

D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code, volume 35. 2001.

Digital Library

[9]

M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon System for Dynamic Detection of Likely Invariants. In Sci. Comput. Program., volume 69, pages 35–45, 2007.

Digital Library

[10]

N. Gruska, A. Wasylkowski, and A. Zeller. Learning from 6,000 Projects: Lightweight Cross-Project Anomaly Detection. In ISSTA’ 10, pages 119–130, 2010.

Digital Library

[11]

M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The WEKA Data Mining Software: An Update. SIGKDD Explor. Newsl., 11(1):10–18, Nov. 2009.

Digital Library

[12]

S. Han, D. R. Wallace, and R. C. Miller. Code Completion from Abbreviated Input. In ASE’ 09, pages 332–343, 2009.

Digital Library

[13]

S. Hangal and M. S. Lam. Tracking Down Software Bugs Using Automatic Anomaly Detection. In ICSE’ 02, pages 291–301, 2002.

Digital Library

[14]

V. J. Hellendoorn, P. T. Devanbu, and A. Bacchelli. Will They Like This?: Evaluating Code Contributions with Language Models. In MSR’ 15, pages 157–167, 2015.

Digital Library

[15]

A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu. On the Naturalness of Software. In ICSE’ 12, pages 837–847, 2012.

Digital Library

[16]

D. Hovemeyer and W. Pugh. Finding Bugs is Easy. ACM Sigplan Notices, 39(12):92–106, 2004.

Digital Library

[17]

C.-H. Hsiao, M. Cafarella, and S. Narayanasamy. Using Web Corpus Statistics for Program Analysis. In OOPSLA ’14, pages 49–65, 2014.

Digital Library

[18]

J. Huang, P. O. Meredith, and G. Rosu. Maximal Sound Predictive Race Detection with Control Flow Abstraction. In PLDI’ 14, pages 337–348, 2014.

Digital Library

[19]

H. Kagdi, M. L. Collard, and J. I. Maletic. An Approach to Mining Call-Usage Patterns with Syntactic Context. In ASE’ 07, pages 457–460, 2007.

Digital Library

[20]

W. M. Khoo, A. Mycroft, and R. Anderson. Rendezvous: A Search Engine for Binary Code. In MSR’ 13, pages 329–338, 2013.

Digital Library

[21]

J. Lawall and D. Lo. An Automated Approach for Finding Variable-constant Pairing Bugs. In ASE’ 10, pages 103–112, 2010.

Digital Library

[22]

Z. Li, S. Lu, S. Myagmar, and Y. Zhou. CP-Miner: A Tool for Finding Copy-paste and Related Bugs in Operating System Code. In OSDI’ 04, pages 20–20, 2004.

Digital Library

[23]

Z. Li and Y. Zhou. PR-Miner: Automatically Extracting Implicit Programming Rules and Detecting Violations in Large Software Code. In FSE’ 05, pages 306–315, 2005.

Digital Library

[24]

B. Liang, P. Bian, Y. Zhang, W. Shi, W. You, and Y. Cai. AntMiner: Mining More Bugs by Reducing Noise Interference. In ICSE’ 16, pages 333–344, 2016.

Digital Library

[25]

H. Liu, Y. Wang, L. Jiang, and S. Hu. PF-Miner: A New Paired Functions Mining Method for Android Kernel in Error Paths. In COMPSAC’ 14, pages 33–42, 2014.

Digital Library

[26]

C. D. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT press, 1999.

Digital Library

[27]

F. Martin et al. Refactoring: Improving the Design of Existing Code. 1999.

Digital Library

[28]

M. Monperrus and M. Mezini. Detecting Missing Method Calls As Violations of the Majority Rule. ACM Trans. Softw. Eng. Methodol., 22(1):7:1–7:25, 2013.

Digital Library

[29]

D. Movshovitz-Attias and W. W. Cohen. Natural Language Models for Predicting Programming Comments. In ACL’ 13, pages 35–40, 2013.

[30]

S. Nessa, M. Abedin, E. Wong, L. Khan, and Y. Qi. Software Fault Localization Using N-gram Analysis. In WASA’ 08, pages 548–559, 2008.

Digital Library

[31]

T. T. Nguyen, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen. A Statistical Semantic Language Model for Source Code. In FSE’ 13, pages 532–542, 2013.

Digital Library

[32]

T. T. Nguyen, H. A. Nguyen, N. H. Pham, J. M. Al-Kofahi, and T. N. Nguyen. Graph-based Mining of Multiple Object Usage Patterns. In FSE’ 09, pages 383–392, 2009.

Digital Library

[33]

Y. Oda, H. Fudaba, G. Neubig, H. Hata, S. Sakti, T. Toda, and S. Nakamura. Learning to Generate Pseudo-code from Source Code Using Statistical Machine Translation. In ASE’ 15, pages 824—829, 2015.

Digital Library

[34]

M. Pradel and T. R. Gross. Automatic Generation of Object Usage Specifications from Large Method Traces. In ASE’ 09, pages 371–382, 2009.

Digital Library

[35]

M. K. Ramanathan, A. Grama, and S. Jagannathan. Path-Sensitive Inference of Function Precedence Protocols. In ICSE’ 07, pages 240–250, 2007.

Digital Library

[36]

B. Ray, V. Hellendoorn, Z. Tu, C. Nguyen, S. Godhane, A. Bacchelli, and P. Devanbu. On the "Naturalness" of Buggy Code. In ICSE’ 16, 2016.

Digital Library

[37]

V. Raychev, M. Vechev, and E. Yahav. Code Completion with Statistical Language Models. In PLDI’ 14, pages 419–428, 2014.

Digital Library

[38]

R. Rosenfield. Two Decades of Statistical Language Modeling: Where Do We Go from Here? 2000.

[39]

S. Saha, J.-P. Lozi, G. Thomas, J. L. Lawall, and G. Muller. Hector: Detecting Resource-release Omission Faults in Error-handling Code for Systems Software. In DSN’ 13, pages 1–12, 2013.

Digital Library

[40]

M. Sampson, L. Zhang, A. Morrison, N. J. Barrowman, T. J. Clifford, R. W. Platt, T. P. Klassen, and D. Moher. An Alternative to the Hand Searching Gold Standard: Validating Methodological Search Filters Using Relative Recall. BMC Medical Research Methodology, 6(1), 2006.

[41]

S. Shoham, E. Yahav, S. Fink, and M. Pistoia. Static Specification Mining Using Automata-Based Abstractions. In ISSTA’ 07, pages 174–184, 2007.

Digital Library

[42]

B. Sun, G. Shu, A. Podgurski, and B. Robinson. Extending Static Analysis by Mining Project-specific Rules. In ICSE’ 12, pages 1054–1063, 2012.

Digital Library

[43]

L. Tan, D. Yuan, G. Krishna, and Y. Zhou. /* iComment: Bugs or Bad Comments? */. In SOSP’ 07, pages 145–158, 2007.

Digital Library

[44]

S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens. @tComment: Testing Javadoc Comments to Detect Comment-code Inconsistencies. In ICST’ 12, pages 260–269, 2012.

Digital Library

[45]

S. Thummalapenta and T. Xie. Mining Exception-Handling Rules as Sequence Association Rules. In ICSE’ 09, pages 496–506, 2009.

Digital Library

[46]

S. Thummalapenta and T. Xie. Alattin: Mining Alternative Patterns for Defect Detection. Automated Software Engineering, 18(3-4):292–323, 2011.

Digital Library

[47]

Z. Tu, Z. Su, and P. Devanbu. On the Localness of Software. In FSE’ 14, pages 269–280, 2014.

Digital Library

[48]

J. Wang, Y. Dang, H. Zhang, K. Chen, T. Xie, and D. Zhang. Mining Succinct and High-coverage API Usage Patterns from Source Code. In MSR’ 13, pages 319–328, 2013.

Digital Library

[49]

A. Wasylkowski and A. Zeller. Mining Temporal Specifications from Object Usage. Automated Software Engineering, 18(3-4):263–292, 2011.

Digital Library

[50]

A. Wasylkowski, A. Zeller, and C. Lindig. Detecting Object Usage Anomalies. In FSE’ 07, pages 35–44, 2007.

Digital Library

[51]

M. White, C. Vendome, M. Linares-Vásquez, and D. Poshyvanyk. Toward Deep Learning Software Repositories. In MSR ’15, pages 334–345, 2015.

Digital Library

[52]

C. Williams and J. Hollingsworth. Automatic Mining of Source Code Repositories to Improve Bug Finding Techniques. In TSE’ 05, volume 31, pages 466–480, 2005.

Digital Library

[53]

C. C. Williams and J. K. Hollingsworth. Recovering System Specific Rules from Software Repositories. In MSR’ 05, pages 1–5, 2005.

Digital Library

[54]

T. Xie and J. Pei. MAPO: Mining API Usages from Open Source Repositories. In MSR’ 06, pages 54–57, 2006.

Digital Library

[55]

Y. Xue, J. Wang, Y. Liu, H. Xiao, J. Sun, and M. Chandramohan. Detection and Classification of Malicious Javascript via Attack Behavior Modelling. In ISSTA’ 15, pages 48–59, 2015.

Digital Library

[56]

J. Yang, D. Evans, D. Bhardwaj, T. Bhat, and M. Das. Perracotta: Mining Temporal API Rules from Imperfect Traces. In ICSE’ 06, pages 282–291, 2006.

Digital Library

[57]

Z. Yu, H. Hu, C. Bai, K.-Y. Cai, and W. Wong. GUI Software Fault Localization Using N-gram Analysis. In HASE’11, pages 325–332, 2011.

Digital Library

Cited By

Stracquadanio GMedya SQuer SPal D(2024)VeriBug: An Attention-Based Framework for Bug Localization in Hardware Designs2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546890(1-2)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546890
Li CZhang JTang YLi ZSun TSpinellis DConstantinou EBacchelli A(2024)Boosting API Misuse Detection via Integrating API Constraints from Multiple SourcesProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644904(14-26)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644904
Huang JNie JGong YYou WLiang BBian PRoychoudhury APaiva AAbreu RStorey M(2024)Raisin: Identifying Rare Sensitive Functions for Bug DetectionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639165(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639165
Show More Cited By

Index Terms

Bugram: bug detection with n-gram language models
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Software functional properties
      1. Formal methods
        Automated static analysis

Recommendations

DeepBugs: a learning approach to name-based bug detection

Natural language elements in source code, e.g., the names of variables and functions, convey useful information. However, most existing bug detection tools ignore this information and therefore miss some classes of bugs. The few existing name-based bug ...
Detect Related Bugs from Source Code Using Bug Information
COMPSAC '10: Proceedings of the 2010 IEEE 34th Annual Computer Software and Applications Conference

Open source projects often maintain open bug repositories during development and maintenance, and the reporters often point out straightly or implicitly the reasons why bugs occur when they submit them. The comments about a bug are very valuable for ...
Are Neural Bug Detectors Comparable to Software Developers on Variable Misuse Bugs?
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

Debugging, that is, identifying and fixing bugs in software, is a central part of software development. Developers are therefore often confronted with the task of deciding whether a given code snippet contains a bug, and if yes, where. Recently, data-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

August 2016

899 pages

ISBN:9781450338455

DOI:10.1145/2970276

General Chair:
David Lo
Singapore Management University, Singapore
,
Program Chairs:
Sven Apel
University of Passau, Germany
,
Sarfraz Khurshid
University of Texas at Austin, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE'16

Sponsor:

SIGAI
SIGSOFT
IEEE-CS

ASE'16: ACM/IEEE International Conference on Automated Software Engineering

September 3 - 7, 2016

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

83
Total Citations
View Citations
1,017
Total Downloads

Downloads (Last 12 months)93
Downloads (Last 6 weeks)6

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Stracquadanio GMedya SQuer SPal D(2024)VeriBug: An Attention-Based Framework for Bug Localization in Hardware Designs2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546890(1-2)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546890
Li CZhang JTang YLi ZSun TSpinellis DConstantinou EBacchelli A(2024)Boosting API Misuse Detection via Integrating API Constraints from Multiple SourcesProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644904(14-26)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644904
Huang JNie JGong YYou WLiang BBian PRoychoudhury APaiva AAbreu RStorey M(2024)Raisin: Identifying Rare Sensitive Functions for Bug DetectionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639165(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639165
Ghaemi HAlizadehsani ZShahraki ACorchado J(2024)Transformers in source code generation: A comprehensive surveyJournal of Systems Architecture10.1016/j.sysarc.2024.103193153(103193)Online publication date: Aug-2024
https://doi.org/10.1016/j.sysarc.2024.103193
Wang QLi ZLiang HPan XLi HLi TLi XLi CGuo S(2024)Graph Confident Learning for Software Vulnerability DetectionEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108296133(108296)Online publication date: Jul-2024
https://doi.org/10.1016/j.engappai.2024.108296
Han JHuang CLiu J(2024)bjCnet: A contrastive learning-based framework for software defect predictionComputers & Security10.1016/j.cose.2024.104024(104024)Online publication date: Jul-2024
https://doi.org/10.1016/j.cose.2024.104024
Caddy JTreude CWagner MBarr E(2024)The role of surprisal in issue trackersEmpirical Software Engineering10.1007/s10664-024-10587-w30:1Online publication date: 23-Nov-2024
https://doi.org/10.1007/s10664-024-10587-w
Yang FZhong FZeng GXiao PZheng W(2024)LineFlowDP: A Deep Learning-Based Two-Phase Approach for Line-Level Defect PredictionEmpirical Software Engineering10.1007/s10664-023-10439-z29:2Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1007/s10664-023-10439-z
Wu XWeng SZheng BZheng WChen XSun X(2024)NG_MDERANK: A software vulnerability feature knowledge extraction method based on N‐gram similarityJournal of Software: Evolution and Process10.1002/smr.2727Online publication date: 27-Aug-2024
https://doi.org/10.1002/smr.2727
Yang Y(2023)Improving Code Completion by Solving Data Inconsistencies in the Source Code with a Hierarchical Language ModelElectronics10.3390/electronics1207157612:7(1576)Online publication date: 27-Mar-2023
https://doi.org/10.3390/electronics12071576
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents