research-article

The evolution of type annotations in python: an empirical study

Authors:

Luca Di Grazia,

Michael PradelAuthors Info & Claims

ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 209 - 220

https://doi.org/10.1145/3540250.3549114

Published: 09 November 2022 Publication History

Abstract

Type annotations and gradual type checkers attempt to reveal errors and facilitate maintenance in dynamically typed programming languages. Despite the availability of these features and tools, it is currently unclear how quickly developers are adopting them, what strategies they follow when doing so, and whether adding type annotations reveals more type errors. This paper presents the first large-scale empirical study of the evolution of type annotations and type errors in Python. The study is based on an analysis of 1,414,936 type annotation changes, which we extract from 1,123,393 commits among 9,655 projects. Our results show that (i) type annotations are getting more popular, and once added, often remain unchanged in the projects for a long time, (ii) projects follow three evolution patterns for type annotation usage -- regular annotation, type sprints, and occasional uses -- and that the used pattern correlates with the number of contributors, (iii) more type annotations help find more type errors (0.704 correlation), but nevertheless, many commits (78.3%) are committed despite having such errors. Our findings show that better developer training and automated techniques for adding type annotations are needed, as most code still remains unannotated, and they call for a better integration of gradual type checking into the development process.

References

[1]

Miltiadis Allamanis, Earl T. Barr, Soline Ducousso, and Zheng Gao. 2020. Typilus: Neural Type Hints. In PLDI.

[2]

Jong-hoon (David) An, Avik Chaudhuri, Jeffrey S. Foster, and Michael Hicks. 2011. Dynamic Inference of Static Types for Ruby. In Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’11). ACM, New York, NY, USA. 459–472. isbn:978-1-4503-0490-0 https://doi.org/10.1145/1926385.1926437

Digital Library

[3]

Christopher Anderson, Paola Giannini, and Sophia Drossopoulou. 2005. Towards Type Inference for JavaScript. In ECOOP 2005 - Object-Oriented Programming, Andrew P. Black (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg. 428–452. isbn:978-3-540-31725-8

[4]

Justus Bogner and Manuel Merkel. 2022. To Type or Not to Type? A Systematic Comparison of the Software Quality of JavaScript and TypeScript Applications on GitHub. arXiv preprint arXiv:2203.11115.

[5]

Lin Chen, Baowen Xu, Tianlin Zhou, and Xiaoyu Zhou. 2009. A Constraint Based Bug Checking Approach for Python. In Computer Software and Applications Conference (COMPSAC). IEEE, 306–311.

Digital Library

[6]

Barthélémy Dagenais and Martin P. Robillard. 2011. Recommending Adaptive Changes for Framework Evolution. ACM Trans. Softw. Eng. Methodol, 20, 4 (2011).

Digital Library

[7]

Julian Dolby, Avraham Shinnar, Allison Allain, and Jenna Reinen. 2018. Ariadne: analysis for machine learning programs. In Proceedings of the 2Nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. 1–10.

Digital Library

[8]

Aryaz Eghbali and Michael Pradel. 2022. DynaPyt: A Dynamic Analysis Framework for Python. In FSE.

[9]

Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. In ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014. 313–324. https://doi.org/10.1145/2642937.2642982

Digital Library

[10]

Beat Fluri, Michael Wuersch, Martin PInzger, and Harald Gall. 2007. Change distilling: Tree differencing for fine-grained source code change extraction. IEEE Transactions on software engineering, 33, 11 (2007), 725–743.

Digital Library

[11]

Michael Furr, Jong-hoon (David) An, and Jeffrey S. Foster. 2009. Profile-guided static typing for dynamic scripting languages. In Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). ACM, 283–300.

[12]

Zheng Gao, Christian Bird, and Earl T. Barr. 2017. To type or not to type: Quantifying detectable bugs in JavaScript. In Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017. 758–769.

[13]

Emanuel Giger, Martin Pinzger, and Harald C Gall. 2011. Comparing fine-grained source code changes and code churn for bug prediction. In Proceedings of the 8th Working Conference on Mining Software Repositories. 83–92.

Digital Library

[14]

Luca Di Grazia, Paul Bredl, and Michael Pradel. 2022. DiffSearch: A Scalable and Precise Search Engine for Code Changes. CoRR, abs/2204.02787 (2022), https://doi.org/10.48550/arXiv.2204.02787 arXiv:2204.02787.

[15]

Felix Grund, Shaiful Alam Chowdhury, Nicholas Bradley, Braxton Hall, and Reid Holmes. 2021. CodeShovel: Constructing Method-Level Source Code Histories. In ICSE.

[16]

Rui Gu, Guoliang Jin, Linhai Song, Linjie Zhu, and Shan Lu. 2015. What change history tells us about thread synchronization. In ESEC/FSE. 426–438.

[17]

Stefan Hanenberg. 2010. An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In Proceedings of the ACM international conference on Object oriented programming systems languages and applications. 22–35.

Digital Library

[18]

Stefan Hanenberg. 2010. An experiment about static and dynamic type systems: doubts about the positive impact of static type systems on development time. In Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA). 22–35.

Digital Library

[19]

Stefan Hanenberg, Sebastian Kleinschmager, Romain Robbes, Éric Tanter, and Andreas Stefik. 2014. An empirical study on the impact of static typing on software maintainability. Empirical Software Engineering, 19, 5 (2014), 1335–1382.

Digital Library

[20]

Masatomo Hashimoto and Akira Mori. 2008. Diff/TS: A Tool for Fine-Grained Structural Change Analysis. In WCRE 2008, Proceedings of the 15th Working Conference on Reverse Engineering, Antwerp, Belgium, October 15-18, 2008. 279–288. https://doi.org/10.1109/WCRE.2008.44

Digital Library

[21]

Masatomo Hashimoto, Akira Mori, and Tomonori Izumida. 2018. Automated patch extraction via syntax- and semantics-aware Delta debugging on source code changes. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018. 598–609. https://doi.org/10.1145/3236024.3236047

Digital Library

[22]

Mostafa Hassan, Caterina Urban, Marco Eilers, and Peter Müller. 2018. MaxSMT-Based Type Inference for Python 3. In International Conference on Computer Aided Verification. 12–19.

[23]

Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. 2018. Deep learning type inference. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 152–162. https://doi.org/10.1145/3236024.3236051

Digital Library

[24]

Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. 2019. DeepJIT: an end-to-end deep learning framework for just-in-time defect prediction. In Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada. 34–45. https://doi.org/10.1109/MSR.2019.00016

Digital Library

[25]

Simon Holm Jensen, Anders Møller, and Peter Thiemann. 2009. Type Analysis for JavaScript. In Symposium on Static Analysis (SAS). Springer, 238–255.

[26]

Wuxia Jin, Dinghong Zhong, Zifan Ding, Ming Fan, and Ting Liu. 2021. Where to Start: Studying Type Annotation Practices in Python. In ASE.

[27]

Ameya Ketkar, Nikolaos Tsantalis, and Danny Dig. 2020. Understanding type changes in java. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 629–641.

Digital Library

[28]

Faizan Khan, Boqi Chen, Daniel Varro, and Shane Mcintosh. 2021. An Empirical Study of Type-Related Defects in Python Projects. IEEE Transactions on Software Engineering.

[29]

Julia Lawall, Quentin Lambert, and Gilles Muller. 2016. Prequel: A Patch-Like Query Language for Commit History Search.

[30]

Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. 2019. NL2Type: Inferring JavaScript function types from natural language information. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. 304–315. https://doi.org/10.1109/ICSE.2019.00045

Digital Library

[31]

Na Meng, Miryung Kim, and Kathryn S. McKinley. 2011. Systematic editing: generating program transformations from an example. In PLDI. 329–342.

[32]

Na Meng, Miryung Kim, and Kathryn S. McKinley. 2013. LASE: locating and applying systematic edits by learning from examples. In ICSE. 502–511.

[33]

Khanh Nguyen and Guoqing Xu. 2013. Cachetor: Detecting Cacheable Data to Remove Bloat. In European Software Engineering Conference and International Symposiumon Foundations of Software Engineering (ESEC/FSE). ACM, 268–278.

Digital Library

[34]

John-Paul Ore, Sebastian Elbaum, Carrick Detweiler, and Lambros Karkazis. 2018. Assessing the type annotation burden. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 190–201.

Digital Library

[35]

Francisco Ortin, Jose Baltasar Garcia Perez-Schofield, and Jose Manuel Redondo. 2015. Towards a static type checker for python. In European Conference on Object-Oriented Programming (ECOOP), Scripts to Programs Workshop, STOP. 15, 1–2.

[36]

Rumen Paletov, Petar Tsankov, Veselin Raychev, and Martin T. Vechev. 2018. Inferring crypto API rules from code changes. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018. 450–464.

[37]

Jibesh Patra and Michael Pradel. 2022. Nalin: Learning from Runtime Behavior to Find Name-Value Inconsistencies in Jupyter Notebooks. In ICSE.

Digital Library

[38]

Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. 2020. TypeWriter: Neural Type Prediction with Search-based Validation. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020. 209–220. https://doi.org/10.1145/3368089.3409715

Digital Library

[39]

Ingkarat Rak-amnouykit, Daniel McCrevan, Ana Milanova, Martin Hirzel, and Julian Dolby. [n.d.]. Python 3 Types in the Wild: A Tale of Two Type Systems.

[40]

Veselin Raychev, Martin T. Vechev, and Andreas Krause. 2015. Predicting Program Properties from "Big Code". In Principles of Programming Languages (POPL). 111–124.

[41]

Brianna M. Ren, John Toman, T. Stephen Strickland, and Jeffrey S. Foster. 2013. The Ruby Type Checker. In Proceedings of the 28th Annual ACM Symposium on Applied Computing (SAC ’13). ACM, New York, NY, USA. 1565–1572. isbn:978-1-4503-1656-9 https://doi.org/10.1145/2480362.2480655

Digital Library

[42]

Gregor Richards, Christian Hammer, Brian Burg, and Jan Vitek. 2011. The Eval That Men Do - A Large-Scale Study of the Use of Eval in JavaScript Applications. In European Conference on Object-Oriented Programming (ECOOP). 52–78.

[43]

Marija Selakovic and Michael Pradel. 2016. Performance Issues and Optimizations in JavaScript: An Empirical Study. In International Conference on Software Engineering (ICSE). 61–72.

Digital Library

[44]

Cristian-Alexandru Staicu and Michael Pradel. 2018. Freezing the Web: A Study of ReDoS Vulnerabilities in JavaScript-based Web Servers. In USENIX Security Symposium. 361–376.

Digital Library

[45]

Kristín Fjóla Tómasdóttir, Mauricio Aniche, and Arie van Deursen. 2017. Why and how JavaScript developers use linters. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 578–589.

[46]

Michael M. Vitousek, Andrew M. Kent, Jeremy G. Siek, and Jim Baker. 2014. Design and evaluation of gradual typing for python. In DLS’14, Proceedings of the 10th ACM Symposium on Dynamic Languages, part of SLASH 2014, Portland, OR, USA, October 20-24, 2014, Andrew P. Black and Laurence Tratt (Eds.). ACM, 45–56. https://doi.org/10.1145/2661088.2661101

Digital Library

[47]

Peter Weiß gerber and Stephan Diehl. 2006. Identifying Refactorings from Source-Code Changes. In International Conference on Automated Software Engineering (ASE). IEEE, 231–240.

[48]

Ming Wen, Rongxin Wu, and Shing-Chi Cheung. 2016. Locus: locating bugs from software changes. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016. 262–273. https://doi.org/10.1145/2970276.2970359

Digital Library

[49]

Rongxin Wu, Ming Wen, Shing-Chi Cheung, and Hongyu Zhang. 2018. ChangeLocator: locate crash-inducing changes based on crash reports. Empirical Software Engineering, 23, 5 (2018), 2866–2900. https://doi.org/10.1007/s10664-017-9567-4

Digital Library

[50]

Xinmeng Xia, Xincheng He, Yanyan Yan, Lei Xu, and Baowen Xu. 2018. An Empirical Study of Dynamic Types for Python Projects. In International Conference on Software Analysis, Testing, and Evolution. 85–100.

[51]

Zhaogui Xu, Peng Liu, Xiangyu Zhang, and Baowen Xu. 2016. Python predictive analysis for bug detection. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, Thomas Zimmermann, Jane Cleland-Huang, and Zhendong Su (Eds.). ACM, 121–132. https://doi.org/10.1145/2950290.2950357

Digital Library

[52]

Zhaogui Xu, Xiangyu Zhang, Lin Chen, Kexin Pei, and Baowen Xu. 2016. Python probabilistic type inference with natural language support. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016. 607–618. https://doi.org/10.1145/2950290.2950343

Digital Library

[53]

Jie M. Zhang, Feng Li, Dan Hao, Meng Wang, Hao Tang, Lu Zhang, and Mark Harman. 2021. A Study of Bug Resolution Characteristics in Popular Programming Languages. IEEE Trans. Software Eng., 47, 12 (2021), 2684–2697. https://doi.org/10.1109/TSE.2019.2961897

Cited By

Guo YChen ZChen LXu WLi YZhou YXu B(2024)Generating Python Type Annotations from Type Inference: How Far Are We?ACM Transactions on Software Engineering and Methodology10.1145/365215333:5(1-38)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3652153
Bouzenia IKrishan BPradel M(2024)DyPyBench: A Benchmark of Executable Python SoftwareProceedings of the ACM on Software Engineering10.1145/36437421:FSE(338-358)Online publication date: 12-Jul-2024
https://doi.org/10.1145/3643742
Venkatesh ASabu SChekkapalli MWang JLi LBodden E(2024)Static analysis driven enhancements for comprehension in machine learning notebooksEmpirical Software Engineering10.1007/s10664-024-10525-w29:5Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1007/s10664-024-10525-w
Show More Cited By

Index Terms

Index terms have been assigned to the content through auto-classification.

Recommendations

An Empirical Study on Dynamic Typing Related Practices in Python Systems
ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

The dynamic typing discipline of Python allows developers to program at a high level of abstraction. However, type related bugs are commonly encountered in Python systems due to the lack of type declaration and static type checking. Especially, the ...
Generating Python Type Annotations from Type Inference: How Far Are We?
In recent years, dynamic languages such as Python have become popular due to their flexibility and productivity. The lack of static typing makes programs face the challenges of fixing type errors, early bug detection, and code understanding. To alleviate ...
ML^F: raising ML to the power of system F
ICFP '03: Proceedings of the eighth ACM SIGPLAN international conference on Functional programming

We propose a type system ML^F that generalizes ML with first-class polymorphism as in System F. Expressions may contain second-order type annotations. Every typable expression admits a principal type, which however depends on type annotations. Principal ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2022

1822 pages

ISBN:9781450394130

DOI:10.1145/3540250

General Chair:
Abhik Roychoudhury
National University of Singapore, Singapore
,
Program Chairs:
Cristian Cadar
Imperial College London, UK
,
Miryung Kim
University of California at Los Angeles, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

ERC

Conference

ESEC/FSE '22

Sponsor:

ESEC/FSE '22: 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 14 - 18, 2022

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
308
Total Downloads

Downloads (Last 12 months)134
Downloads (Last 6 weeks)10

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Guo YChen ZChen LXu WLi YZhou YXu B(2024)Generating Python Type Annotations from Type Inference: How Far Are We?ACM Transactions on Software Engineering and Methodology10.1145/365215333:5(1-38)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3652153
Bouzenia IKrishan BPradel M(2024)DyPyBench: A Benchmark of Executable Python SoftwareProceedings of the ACM on Software Engineering10.1145/36437421:FSE(338-358)Online publication date: 12-Jul-2024
https://doi.org/10.1145/3643742
Venkatesh ASabu SChekkapalli MWang JLi LBodden E(2024)Static analysis driven enhancements for comprehension in machine learning notebooksEmpirical Software Engineering10.1007/s10664-024-10525-w29:5Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1007/s10664-024-10525-w
Shivarpatna Venkatesh AWang JLi LBodden E(2023)Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00044(391-401)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00044

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents