article

/icomment: bugs or bad comments?/

Authors:

Yuanyuan ZhouAuthors Info & Claims

ACM SIGOPS Operating Systems Review, Volume 41, Issue 6

Pages 145 - 158

https://doi.org/10.1145/1323293.1294276

Published: 14 October 2007 Publication History

Abstract

Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information regarding a program's semantic behavior. As software evolves, they can easily grow out-of-sync, indicating two problems: (1) bugs -the source code does not follow the assumptions and requirements specified by correct program comments; (2) bad comments - comments that are inconsistent with correct code, which can confuse and mislead programmers to introduce bugs in subsequent versions. Unfortunately, as most comments are written in natural language, no solution has been proposed to automatically analyze commentsand detect inconsistencies between comments and source code. This paper takes the first step in automatically analyzing commentswritten in natural language to extract implicit program rulesand use these rules to automatically detect inconsistencies between comments and source code, indicating either bugs or bad comments. Our solution, iComment, combines Natural Language Processing(NLP), Machine Learning, Statistics and Program Analysis techniques to achieve these goals. We evaluate iComment on four large code bases: Linux, Mozilla, Wine and Apache. Our experimental results show that iComment automatically extracts 1832 rules from comments with 90.8-100% accuracy and detects 60 comment-code inconsistencies, 33 newbugs and 27 bad comments, in the latest versions of the four programs. Nineteen of them (12 bugs and 7 bad comments) have already been confirmed by the corresponding developers while the others are currently being analyzed by the developers.

Supplementary Material

JPG File (1294276.jpg)

Download
14.07 KB

index.html (index.html)

Slides from the presentation

Download
.93 KB

ZIP File (p145-slides.zip)

Supplemental material for /*icomment: bugs or bad comments?*/

Download
21.57 MB

Audio only (1294276.mp3)

Download
11.93 MB

Video (1294276.mp4)

Download
164.70 MB

References

[1]

C# XML comments let you build documentation directly from your Visual Studio .NET source files. http://msdn.microsoft.com/msdnmag/issues/02/06/XMLC/.

[2]

CoNLL-2000 shared task web page -- with data, software and systems' outputs availble. http://www.cnts.ua.ac.be/conll/.

[3]

Doxygen -- source code documentation generator tool. http://www.stack.nl/ dimitri/doxygen/.

[4]

FreeBSD problem report database. http://www.freebsd.org/support/bugreports.html.

[5]

Java annotations. http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html.

[6]

Javadoc tool. http://java.sun.com/j2se/javadoc/.

[7]

Lock_Lint -- Static data race and deadlock detection tool for C. http://developers.sun.com/sunstudio/articles/locklint.html.

[8]

MSDN run-time library reference -- SAL annotations. http://msdn2.microsoft.com/en--us/library/ms235402.aspx.

[9]

NLP tools. http://l2r.cs.uiuc.edu/~cogcomp/tools.php.

[10]

RDoc -- documentation from Ruby source files. http://rdoc.sourceforge.net/.

[11]

Sparse -- A semantic parser for C. http://www.kernel.org/pub/software/devel/sparse/.

[12]

M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003.

Digital Library

[13]

J.-D. Choi, M. Burke, and P. Carini. Efficient flow-sensitive interprocedural computation of pointer-induced aliases and side effects. In Proceedings of the 20th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 1993.

Digital Library

[14]

D. L. Detlefs, K. R. M. Leino, G. Nelson, and J. B. Saxe. Extended static checking, SRC research report 159. ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/researchreports/SRC-159.ps.

[15]

D. R. Engler and K. Ashcraft. RacerX: Effective, static detection of race conditions and deadlocks. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003.

Digital Library

[16]

D. R. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, 2001.

Digital Library

[17]

M. D. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In Proceedings of the 22nd International Conference on Software Engineering, 2000.

Digital Library

[18]

D. Evans and D. Larochelle. Improving security using extensible lightweight static analysis. IEEE Software, 2002.

Digital Library

[19]

Y. Even-Zohar and D. Roth. A sequential model for multi class classification. In Proceedings of the Conference on Empirical Methods for Natural Language Processing, 2001.

[20]

S. Hallem, B. Chelf, Y. Xie, and D. R. Engler. A system and language for building system-specific, static analyses. In Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation.

Digital Library

[21]

W. E. Howden. Comments analysis and programming errors. IEEE Transactions on Software Engineering, 1990.

Digital Library

[22]

Z. M. Jiang and A. E. Hassan. Examining the evolution of code comments in PostgreSQL. In Proceedings of the 2006 International Workshop on Mining Software Repositories.

Digital Library

[23]

S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX Annual Technical Conference, 2005.

Digital Library

[24]

T. Kremenek, P. Twohey, G. Back, A. Y. Ng, and D. R. Engler. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th USENIX Symposium on Operating System Design and Implementation, 2006.

Digital Library

[25]

T. Li, C. Ellis, A. Lebeck, and D. Sorin. On-demand and semantic-free dynamic deadlock detection with speculative execution. In USENIX Annual Technical Conference, 2005.

Digital Library

[26]

Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In Proceedings of the 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2005.

Digital Library

[27]

C. D. Manning and H. Schütze. Foundations Of Statistical Natural Language Processing. The MIT Press, 2001.

Digital Library

[28]

T. Mitchell. Machine Learning. McGraw Hill, 1997.

Digital Library

[29]

M. Musuvathi, D. Y. W. Park, A. Chou, D. R. Engler, and D. L. Dill. CMC: A pragmatic approach to model checking real code. In Proceedingts of the 5th Symposium on Operating Systems Design and Implementation, 2002.

Digital Library

[30]

A. C. Myers and B. Liskov. Protecting privacy using the decentralized label model. ACM Trans. Softw. Eng. Methodol., 2000.

Digital Library

[31]

S. E. Perl and W. E. Weihl. Performance assertion checking. In Proceedings of the 14th ACM Symposium on Operating Systems Principles, 1993.

Digital Library

[32]

V. Punyakanok and D. Roth. The use of classifiers in sequential inference. In Proceedings of the Conference on Advances in Neural Information Processing Systems, 2001.

[33]

V. Punyakanok, D. Roth, and W. Yih. The necessity of syntactic parsing for semantic role labeling. In Proceedings of the International Joint Conference on Artificial Intelligence, 2005.

Digital Library

[34]

R. J. Quilan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.

Digital Library

[35]

A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 1996.

[36]

K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. ESC/Java user's manual, SRC technical note 2000-002. http://gatekeeper.dec.com/pub/DEC/SRC/technicalnotes/abstracts/src-tn-2000-002.html.

[37]

S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 1997.

Digital Library

[38]

B. Steensgaard. Points-to analysis in almost linear time. In Proceedings of the 23rd Annual ACM SIGPLAN--SIGACT Symposium on Principles of Programming Languages, 1996.

Digital Library

[39]

N. Sterling. WARLOCK -- A static data race analysis tool. In USENIX Winter Technical Conference, pages 97--106, 1993.

[40]

S. Teufel and M. Moens. Summarizing scientific articles -- experiments with relevance and rhetorical status. Computational Linguistics, 2002.

Digital Library

[41]

I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques (2nd Ed.). Morgan Kaufmann, 2005.

Digital Library

[42]

S. N. Woodfield, H. E. Dunsmore, and V. Y. Shen. The effect of modularization and comments on program comprehension. In Proceedings of the 5th International Conference on Software Engineering, 1981.

Digital Library

[43]

A. Yaar, A. Perrig, and D. X. Song. Pi: A path identification mechanism to defend against DDoS attack. In IEEE Symposium on Security and Privacy, 2003.

Digital Library

[44]

A. T. T. Ying, J. L. Wright, and S. Abrams. Source code that talks: An exploration of eclipse task comments and their implication to repository mining. In Proceedings of the 2005 International Workshop on Mining Software Repositories.

Digital Library

[45]

C. Zhai, A. Velivelli, and B. Yu. A cross-collection mixture model for comparative text mining. In Proceedings of the 2004 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining.

Digital Library

[46]

F. Zhou, J. Condit, Z. Anderson, I. Bagrak, R. Ennals, M. Harren, G. Necula, and E. Brewer. SafeDrive: Safe and recoverable extensions using language-based techniques. In Proceedings of the 7th Symposium on Operating System Design and Implementation, 2006.

Digital Library

Cited By

Jabrayilzade EYurtoğlu ATüzün E(2024)Taxonomy of inline code comment smellsEmpirical Software Engineering10.1007/s10664-023-10425-529:3Online publication date: 3-Apr-2024
https://doi.org/10.1007/s10664-023-10425-5
Pearce HTan BAhmad BKarri RDolan-Gavitt B(2023)Examining Zero-Shot Vulnerability Repair with Large Language Models2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179420(2339-2356)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179420
Pearce HTan BAhmad BKarri RDolan-Gavitt B(2023)Examining Zero-Shot Vulnerability Repair with Large Language Models2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179324(2339-2356)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179324
Show More Cited By

Index Terms

/*icomment: bugs or bad comments?*/
1. General and reference
  1. Cross-computing tools and techniques
    1. Reliability
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Documentation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

/*icomment: bugs or bad comments?*/
SOSP '07: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles

Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information ...
The Secret Life of Commented-Out Source Code
ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

Source code commenting is a common practice to improve code comprehension in software development. While comments often consist of descriptive natural language, surprisingly, there exists a non-trivial portion of comments that are actually code ...
AutoComment: mining question and answer sites for automatic comment generation
ASE '13: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering

Code comments improve software maintainability. To address the comment scarcity issue, we propose a new automatic comment generation approach, which mines comments from a large programming Question and Answer (Q&A) site. Q&A sites allow programmers to ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGOPS Operating Systems Review

ACM SIGOPS Operating Systems Review Volume 41, Issue 6

SOSP '07

December 2007

363 pages

ISSN:0163-5980

DOI:10.1145/1323293

Issue’s Table of Contents

SOSP '07: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles
October 2007
378 pages
ISBN:9781595935915
DOI:10.1145/1294261
General Chair:
Thomas C. Bressoud
Denison University, USA
,
Program Chair:
M. Frans Kaashoek
Massachusetts Institute of Technology, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2007

Published in SIGOPS Volume 41, Issue 6

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

227
Total Citations
View Citations
1,815
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)9

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jabrayilzade EYurtoğlu ATüzün E(2024)Taxonomy of inline code comment smellsEmpirical Software Engineering10.1007/s10664-023-10425-529:3Online publication date: 3-Apr-2024
https://doi.org/10.1007/s10664-023-10425-5
Pearce HTan BAhmad BKarri RDolan-Gavitt B(2023)Examining Zero-Shot Vulnerability Repair with Large Language Models2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179420(2339-2356)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179420
Pearce HTan BAhmad BKarri RDolan-Gavitt B(2023)Examining Zero-Shot Vulnerability Repair with Large Language Models2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179324(2339-2356)Online publication date: May-2023
https://doi.org/10.1109/SP46215.2023.10179324
Yao KOliva GHassan AAsaduzzaman MMalton AWalenstein A(2023)Finding associations between natural and computer languagesJournal of Systems and Software10.1016/j.jss.2023.111651201:COnline publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111651
Rani PBlasi AStulova NPanichella SGorla ANierstrasz O(2023)A decade of code comment quality assessment: A systematic literature reviewJournal of Systems and Software10.1016/j.jss.2022.111515195(111515)Online publication date: Jan-2023
https://doi.org/10.1016/j.jss.2022.111515
Nassif MHernandez ASridharan ARobillard M(2022)Generating Unit Tests for DocumentationIEEE Transactions on Software Engineering10.1109/TSE.2021.308708748:9(3268-3279)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TSE.2021.3087087
Digkas GChatzigeorgiou AAmpatzoglou AAvgeriou P(2022)Can Clean New Code Reduce Technical Debt Density?IEEE Transactions on Software Engineering10.1109/TSE.2020.303255748:5(1705-1721)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1109/TSE.2020.3032557
Yu ZFahid FTu HMenzies T(2022)Identifying Self-Admitted Technical Debts With Jitterbug: A Two-Step ApproachIEEE Transactions on Software Engineering10.1109/TSE.2020.303140148:5(1676-1691)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1109/TSE.2020.3031401
Verdi MSami AAkhondali JKhomh FUddin GMotlagh A(2022)An Empirical Study of C++ Vulnerabilities in Crowd-Sourced Code ExamplesIEEE Transactions on Software Engineering10.1109/TSE.2020.302366448:5(1497-1514)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1109/TSE.2020.3023664
Huber GBogenberger Kvan Lint H(2022)Optimization of Charging Strategies for Battery Electric Vehicles Under UncertaintyIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2020.302762523:2(760-776)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1109/TITS.2020.3027625
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents