research-article

C2S: translating natural language comments to formal program specifications

Authors:

Xiangyu ZhangAuthors Info & Claims

ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 25 - 37

https://doi.org/10.1145/3368089.3409716

Published: 08 November 2020 Publication History

Abstract

Formal program specifications are essential for various software engineering tasks, such as program verification, program synthesis, code debugging and software testing. However, manually inferring formal program specifications is not only time-consuming but also error-prone. In addition, it requires substantial expertise. Natural language comments contain rich semantics about behaviors of code, making it feasible to infer program specifications from comments. Inspired by this, we develop a tool, named C2S, to automate the specification synthesis task by translating natural language comments into formal program specifications. Our approach firstly constructs alignments between natural language word and specification tokens from existing comments and their corresponding specifications. Then for a given method comment, our approach assembles tokens that are associated with words in the comment from the alignments into specifications guided by specification syntax and the context of the target method. Our tool successfully synthesizes 1,145 specifications for 511 methods of 64 classes in 5 different projects, substantially outperforming the state-of-the-art. The generated specifications are also used to improve a number of software engineering tasks like static taint analysis, which demonstrates the high quality of the specifications.

Supplementary Material

Auxiliary Teaser Video (fse20main-p365-p-teaser.mp4)

Translating Natural Language Comments to Formal Program Specifications

Download
3.88 MB

Auxiliary Presentation Video (fse20main-p365-p-video.mp4)

Translating Natural Language Comments to Formal Program Specifications

Download
26.42 MB

References

[1]

2020. Apache Commons Collections. https://commons.apache.org/proper/ commons-collections/.

[2]

2020. C2S Specifications. https://c2s-fse.github.io/C2S/.

[3]

2020. GraphStream. http://graphstream-project. org/.

[4]

2020. Guava. https://opensource.google.com/projects/guava/.

[5]

2020. Javadoc Style. https://www.oracle.com/technetwork/articles/java/index137868.html.

[6]

2020. JDK. https://www.oracle.com/technetwork/java/javase/downloads/index. html.

[7]

2020. JGraphT. https://jgrapht.org/.

[8]

2020. JML Specification Examples. http://www.eecs.ucf.edu/~leavens/JML/ examples.shtml.

[9]

2020. Randoop. https://randoop.github.io/randoop/.

[10]

Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo MK Martin, Mukund Raghothaman, Sanjit A Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-guided synthesis. In 2013 Formal Methods in Computer-Aided Design. IEEE, 1-8.

[11]

W Appel Andrew and P Jens. 2002. Modern compiler implementation in Java.

[12]

Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. Acm Sigplan Notices 49, 6 ( 2014 ), 259-269.

Digital Library

[13]

Angello Astorga, Siwakorn Srisakaokul, Xusheng Xiao, and Tao Xie. 2018. PreInfer: Automatic Inference of Preconditions via Symbolic Analysis. In 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE, 678-689.

[14]

Osbert Bastani, Saswat Anand, and Alex Aiken. 2015. Specification inference using context-free language reachability. In ACM SIGPLAN Notices, Vol. 50. ACM, 553-566.

Digital Library

[15]

Arianna Blasi, Alberto Gofi, Konstantin Kuznetsov, Alessandra Gorla, Michael D Ernst, Mauro Pezzè, and Sergio Delgado Castellanos. 2018. Translating code comments to procedure specifications. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 242-253.

Digital Library

[16]

Manuel Costa, Miguel Castro, Lidong Zhou, Lintao Zhang, and Marcus Peinado. 2007. Bouncer: Securing software by blocking bad input. ACM SIGOPS Operating Systems Review 41, 6 ( 2007 ), 117-130.

Digital Library

[17]

Patrick Cousot, Radhia Cousot, Manuel Fähndrich, and Francesco Logozzo. 2013. Automatic inference of necessary preconditions. In International Workshop on Verification, Model Checking, and Abstract Interpretation. Springer, 128-148.

Digital Library

[18]

Christoph Csallner, Nikolai Tillmann, and Yannis Smaragdakis. 2008. DySy: Dynamic Symbolic Execution for Invariant Inference. In Proceedings of the 30th International Conference on Software Engineering. ICSE 2008. IEEE.

Digital Library

[19]

Michael D Ernst, Jake Cockrell, William G Griswold, and David Notkin. 2001. Dynamically discovering likely program invariants to support program evolution. IEEE Transactions on Software Engineering 27, 2 ( 2001 ), 99-123.

Digital Library

[20]

Alberto Gofi, Alessandra Gorla, Michael D Ernst, and Mauro Pezzè. 2016. Automatic generation of oracles for exceptional behaviors. In Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, 213-224.

Digital Library

[21]

Jianjun Huang, Xiangyu Zhang, and Lin Tan. 2016. Detecting sensitive data disclosure via bi-directional text correlation analysis. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 169-180.

Digital Library

[22]

James A Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering. ICSE 2002. IEEE, 467-477.

[23]

Douglas Kramer. 1999. API documentation from source code comments: a case study of Javadoc. In Proceedings of the 17th annual international conference on Computer documentation. 147-153.

Digital Library

[24]

Viktor Kuncak, Mikaël Mayer, Ruzica Piskac, and Philippe Suter. 2010. Complete functional synthesis. In ACM Sigplan Notices, Vol. 45. ACM, 316-329.

Digital Library

[25]

Andreas Leitner, Ilinca Ciupa, Manuel Oriol, Bertrand Meyer, and Arno Fiva. 2007. Contract driven development= test driven development-writing test cases. In Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 425-434.

Digital Library

[26]

Walid Maalej and Martin P Robillard. 2013. Patterns of knowledge in API reference documentation. IEEE Transactions on Software Engineering 39, 9 ( 2013 ), 1264-1282.

Digital Library

[27]

Bertrand Meyer, Ilinca Ciupa, Andreas Leitner, and Lisa Ling Liu. 2007. Automatic testing of object-oriented software. In International Conference on Current Trends in Theory and Practice of Computer Science. Springer, 114-129.

Digital Library

[28]

Manish Motwani and Yuriy Brun. 201. Automatically Generating Precise Oracles from Structured Natural Language Specifications. In Proceedings of the 41th International Conference on Software Engineering (ICSE'19).

[29]

Hoan Anh Nguyen, Robert Dyer, Tien N Nguyen, and Hridesh Rajan. 2014. Mining preconditions of APIs in large-scale code corpus. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 166-177.

Digital Library

[30]

Jeremy W Nimmer and Michael D Ernst. 2002. Automatic generation of program specifications. ACM SIGSOFT Software Engineering Notes 27, 4 ( 2002 ), 229-239.

[31]

Yoann Padioleau, Lin Tan, and Yuanyuan Zhou. 2009. Listening to programmers Taxonomies and characteristics of comments in operating system code. In Proceedings of the 31st International Conference on Software Engineering. IEEE Computer Society, 331-341.

Digital Library

[32]

Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring method specifications from natural language API descriptions. In Proceedings of the 34th International Conference on Software Engineering. IEEE Press, 815-825.

Digital Library

[33]

Hung Phan, Hoan Anh Nguyen, Tien N Nguyen, and Hridesh Rajan. 2017. Statistical learning for inference between implementations and documentation. In Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track. IEEE Press, 27-30.

Digital Library

[34]

M.F. Porter. 1980. An algorithm for sufix stripping. Program 14, 3 ( 1980 ), 130-137.

[35]

Murali Krishna Ramanathan, Ananth Grama, and Suresh Jagannathan. 2007. Static specification inference using predicate mining. In ACM SIGPLAN Notices, Vol. 42. ACM, 123-134.

Digital Library

[36]

Mohamed Nassim Seghir and Daniel Kroening. 2013. Counterexample-guided precondition inference. In European Symposium on Programming. Springer, 451-471.

Digital Library

[37]

C. Silva and B. Ribeiro. 2003. The importance of stop word removal on recall values in text categorization. In Proceedings of the International Joint Conference on Neural Networks. IEEE.

[38]

Saurabh Srivastava, Sumit Gulwani, and Jefrey S Foster. 2010. From program verification to program synthesis. In ACM Sigplan Notices, Vol. 45. ACM, 313-326.

Digital Library

[39]

Jingyi Su, Mohd Arafat, and Robert Dyer. 2018. Poster: Using Consensus to Automatically Infer Post-conditions. ( 2018 ).

[40]

Lin Tan. 2009. Leveraging Code Comments To Improve Software Reliability. Ph.D. Dissertation. University of Illinois at Urbana-Champaign.

[41]

Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /* iComment: Bugs or bad comments?*. In ACM SIGOPS Operating Systems Review, Vol. 41. ACM, 145-158.

Digital Library

[42]

Lin Tan, Yuanyuan Zhou, and Yoann Padioleau. 2011. aComment: mining annotations from comments and code to detect interrupt related concurrency bugs. In Software Engineering (ICSE), 2011 33rd International Conference on. IEEE, 11-20.

Digital Library

[43]

Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T Leavens. 2012. @tcomment: Testing javadoc comments to detect comment-code inconsistencies. In Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on. IEEE, 260-269.

[44]

Niki Vazou, Eric L Seidel, Ranjit Jhala, Dimitrios Vytiniotis, and Simon PeytonJones. 2014. Refinement types for Haskell. In ACM SIGPLAN Notices, Vol. 49. ACM, 269-282.

Digital Library

[45]

Willem Visser, Klaus Havelund, Guillaume Brat, SeungJoon Park, and Flavio Lerda. 2003. Model checking programs. Automated software engineering 10, 2 ( 2003 ), 203-232.

[46]

Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failureinducing input. IEEE Transactions on Software Engineering 28, 2 ( 2002 ), 183-200.

Digital Library

[47]

Juan Zhai, Jianjun Huang, Shiqing Ma, Xiangyu Zhang, Lin Tan, Jianhua Zhao, and Feng Qin. 2016. Automatic model generation from documentation for Java API functions. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 380-391.

Digital Library

[48]

Juan Zhai, Bin Li, Zhenhao Tang, Jianhua Zhao, and Xuandong Li. 2016. Precondition Calculation for Loops Iterating over Data Structures. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 132-143.

[49]

Juan Zhai, Hanfei Wang, and Jianhua Zhao. 2014. Post-condition-directed invariant inference for loops over data structures. In Software Security and ReliabilityCompanion (SERE-C), 2014 IEEE Eighth International Conference on. IEEE, 204-212.

Digital Library

[50]

Juan Zhai, Hanfei Wang, and Jianhua Zhao. 2015. Assertion-directed precondition synthesis for loops over data structures. In International Symposium on Dependable Software Engineering: Theories, Tools, and Applications. Springer, 258-274.

Digital Library

[51]

Juan Zhai, Xiangzhe Xu, Yu Shi, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2020. CPC: Automatically classifying and propagating natural language comments via program analysis. In Software Engineering (ICSE), 2020 IEEE/ACM 42nd International Conference on. IEEE, 1359-1371.

Digital Library

[52]

Shiyu Zhang, Juan Zhai, Bu Lei, Wang Linzhang, and Xuandong Li. 2020. Automated Generation of LTL Specifications For Smart Home IoT Using Natural Language. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE.

[53]

Hao Zhong, Lu Zhang, Tao Xie, and Hong Mei. 2009. Inferring resource speciifcations from natural language API documentation. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, 307-318.

Digital Library

[54]

Yu Zhou, Ruihang Gu, Taolue Chen, Zhiqiu Huang, Sebastiano Panichella, and Harald Gall. 2017. Analyzing APIs documentation and code to detect directive defects. In Proceedings of the 39th International Conference on Software Engineering.

Digital Library

[55]

IEEE Press, 27-37.

Cited By

Wang CZhang JWu RZhang C(2024)DAInfer: Inferring API Aliasing Specifications from Library Documentation via Neurosymbolic OptimizationProceedings of the ACM on Software Engineering10.1145/36608161:FSE(2469-2492)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660816
Li YNguyen TWang SYadavally ARoychoudhury APaiva AAbreu RStorey M(2024)Poirot: Deep Learning for API Misuse DetectionProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3643080(302-303)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3643080
Eladawy HLe Goues CBrun YRoychoudhury APaiva AAbreu RStorey M(2024)Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639095(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639095
Show More Cited By

Index Terms

C2S: translating natural language comments to formal program specifications
1. Software and its engineering

Recommendations

From formal specifications to natural language: a case study
ASE '97: Proceedings of the 12th international conference on Automated software engineering (formerly: KBSE)

Because software specifications often serve as a formal contract between the developer and the customer, systems have been proposed that help the software client better understand specifications by automatically paraphrasing them in natural language. ...
Trustworthy Formal Natural Language Specifications
Onward! 2023: Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software

Interactive proof assistants are computer programs carefully constructed to check a human-designed proof of a mathematical claim with high confidence in the implementation. However, this only validates truth of a formal claim, which may have been ...
Generating Maude formal specifications from AUML diagrams
Selected papers from the International Conference on Computer Science, Software Engineering, Information Technology, e-Business, and Applications, 2004

In this paper, we present a formal and systematic approach allowing translating the specification of the interactions between agents, described using AUML formalism, in a Maude specification. Based on rewriting logic, the formal and object-oriented ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2020

1703 pages

ISBN:9781450370431

DOI:10.1145/3368089

General Chair:
Prem Devanbu
University of California at Davis, USA
,
Program Chairs:
Myra Cohen
Iowa State University, USA
,
Thomas Zimmermann
Microsoft Research, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '20

Sponsor:

SIGSOFT

ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 8 - 13, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
632
Total Downloads

Downloads (Last 12 months)99
Downloads (Last 6 weeks)17

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang CZhang JWu RZhang C(2024)DAInfer: Inferring API Aliasing Specifications from Library Documentation via Neurosymbolic OptimizationProceedings of the ACM on Software Engineering10.1145/36608161:FSE(2469-2492)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660816
Li YNguyen TWang SYadavally ARoychoudhury APaiva AAbreu RStorey M(2024)Poirot: Deep Learning for API Misuse DetectionProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3643080(302-303)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3643080
Eladawy HLe Goues CBrun YRoychoudhury APaiva AAbreu RStorey M(2024)Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639095(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639095
Wu JHe HGao KXiao WLi JZhou M(2024)A comprehensive analysis of challenges and strategies for software release notes on GitHubEmpirical Software Engineering10.1007/s10664-024-10486-029:5Online publication date: 20-Jun-2024
https://doi.org/10.1007/s10664-024-10486-0
Wen CCao JSu JXu ZQin SHe MLi HCheung STian C(2024)Enchanting Program Specification Synthesis by Large Language Models Using Static Analysis and Program VerificationComputer Aided Verification10.1007/978-3-031-65630-9_16(302-328)Online publication date: 25-Jul-2024
https://doi.org/10.1007/978-3-031-65630-9_16
Sumanth Prabhu SFedyukovich GD’Souza D(2024)Maximal Quantified Precondition Synthesis for Linear Array LoopsProgramming Languages and Systems10.1007/978-3-031-57267-8_10(245-274)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57267-8_10
Wang DLi YZhang ZChen KCalandrino JTroncoso C(2023)CarpetFuzzProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620345(1919-1936)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620345
Jabbarvand RTizpaz-Niari SBarr EChandra S(2023)Summary of the 1st Interpretability and Robustness in Neural Software Engineering (InteNSE 2023)ACM SIGSOFT Software Engineering Notes10.1145/3635439.363544649:1(30-33)Online publication date: 27-Dec-2023
https://doi.org/10.1145/3635439.3635446
Alonso JSegura SRuiz-Cortés AJust RFraser G(2023)AGORA: Automated Generation of Test Oracles for REST APIsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598114(1018-1030)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598114
Liu ZLiu KXia XYang XJust RFraser G(2023)Towards More Realistic Evaluation for Neural Test Oracle GenerationProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598080(589-600)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598080
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents