research-article

An empirical study of the influence of static type systems on the usability of undocumented software

Authors:

Stefan Hanenberg,

Andreas StefikAuthors Info & Claims

ACM SIGPLAN Notices, Volume 47, Issue 10

Pages 683 - 702

https://doi.org/10.1145/2398857.2384666

Published: 19 October 2012 Publication History

Abstract

Abstract Although the study of static and dynamic type systems plays a major role in research, relatively little is known about the impact of type systems on software development. Perhaps one of the more common arguments for static type systems in languages such as Java or C++ is that they require developers to annotate their code with type names, which is thus claimed to improve the documentation of software. In contrast, one common argument against static type systems is that they decrease flexibility, which may make them harder to use. While these arguments are found in the literature, rigorous empirical evidence is lacking. We report on a controlled experiment where 27 subjects performed programming tasks on an undocumented API with a static type system (requiring type annotations) as well as a dynamic type system (which does not). Our results show that for some tasks, programmers had faster completion times using a static type system, while for others, the opposite held. We conduct an exploratory study to try and theorize why.

References

[1]

Bird, R., and Wadler, P. An introduction to functional programming. Prentice Hall International (UK) Ltd., Hertfordshire, UK, UK, 1988.

Digital Library

[2]

Bortz, J. Statistik: für Human- und Sozialwissenschaftler, 6., vollst. überarb. u. aktualisierte aufl. ed. Springer, September 2005.

[3]

Brooks, R. E. Studying programmer behavior experimentally: the problems of proper methodology. Commun. ACM 23 (April 1980), 207--213.

Digital Library

[4]

Bruce, K. B. Foundations of object-oriented languages: types and semantics. MIT Press, Cambridge, MA, USA, 2002.

Digital Library

[5]

Curtis, B. Five paradigms in the psychology of programming. In Handbook of Human-Computer Interaction, M. Helander, Ed. Elsevier (North-Holland), 1988, pp. 87--106.

[6]

Daly, M. T., Sazawal, V., and Foster, J. S. Work in progress: an empirical study of static typing in ruby. Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU),Orlando, Florida, October 2009 (2009).

[7]

Fenton, N. E., and Pfleeger, S. L. Software Metrics: A Rigorous and Practical Approach. PWS Publishing Co., Boston, MA, USA, 1998.

Digital Library

[8]

Gannon, J. D. An experimental evaluation of data type conventions. Commun. ACM 20, 8 (1977), 584--595.

Digital Library

[9]

Gravetter, F., and Wallnau, L. Statistics for the Behavioral Sciences. Cengage Learning, 2008.

[10]

Hanenberg, S. An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In Proceedings of the ACM international conference on Object oriented programming systems languages and applications (New York, NY, USA, 2010), OOPSLA '10, ACM, pp. 22--35.

Digital Library

[11]

Hanenberg, S. Faith, hope, and love: An essay on software science's neglect of human factors. In Proceedings of the ACM international conference on Object oriented programming systems languages and applications (Reno/Tahoe, Nevada, USA, October 2010), OOPSLA '10, pp. 933--946.

Digital Library

[12]

Hanenberg, S. A chronological experience report from an initial experiment series on static type systems. In 2nd Workshop on Empirical Evaluation of Software Composition Techniques (ESCOT) (Lancaster, UK, 2011).

[13]

Juristo, N., and Moreno, A. M. Basics of Software Engineering Experimentation. Springer, 2001.

Digital Library

[14]

Kawrykow, D., and Robillard, M. P. Improving API usage through automatic detection of redundant code. In ASE 2009, 24th IEEE/ACM International Conference on Automated Software Engineering, Auckland, New Zealand, November 16--20, 2009 (2009), IEEE Computer Society, pp. 111--122.

Digital Library

[15]

Kleinschmager, S., Hanenberg, S., Robbes, R., Tanter, É., and Stefik, A. Do static type systems improve the maintainability of software systems? An empirical study. In IEEE 20th International Conference on Program Comprehension, ICPC 2012, Passau, Germany, June 11--13, 2012 (2012), pp. 153--162.

[16]

Koenig, D., Glover, A., King, P., Laforge, G., and Skeet, J. Groovy in Action. Manning Publications Co., Greenwich, CT, USA, 2007.

Digital Library

[17]

McConnell, S. What does 10x mean? Measuring variations in programmer productivity. In Making Software: What Really Works, and Why We Believe It, A. Oram and G. Wilson, Eds., O'Reilly Series. O'Reilly Media, 2010, pp. 567--575.

[18]

Pierce, B. C. Types and programming languages. MIT Press, Cambridge, MA, USA, 2002.

Digital Library

[19]

Pottier, F., and Rémy, D. The essence of ML type inference. In Advanced Topics in Types and Programming Languages, B. C. Pierce, Ed. MIT Press, 2005, ch. 10, pp. 389--489.

[20]

Prechelt, L. An empirical comparison of seven programming languages. IEEE Computer 33 (2000), 23--29.

Digital Library

[21]

Prechelt, L. Kontrollierte Experimente in der Softwaretechnik. Springer, Berlin, March 2001.

[22]

Prechelt, L., and Tichy, W. F. A controlled experiment to assess the benefits of procedure argument type checking. IEEE Trans. Softw. Eng. 24, 4 (1998), 302--312.

Digital Library

[23]

Rice, J. A. Mathematical Statistics and Data Analysis. Duxbury Press, Apr. 2001.

[24]

Robillard, M. P. What makes APIs hard to learn? answers from developers. IEEE Software 26, 6 (2009), 27--34.

Digital Library

[25]

Shneiderman, B. Software Psychology: Human Factors in Computer and Information Systems. Winthrop Publishers, August 1980.

Digital Library

[26]

Siek, J. G., and Taha, W. Gradual typing for objects. In ECOOP 2007 - Object-Oriented Programming, 21st European Conference, Berlin, Germany, July 30 - August 3, 2007, Proceedings (2007), vol. 4609 of Lecture Notes in Computer Science, Springer, pp. 2--27.

Digital Library

[27]

Steinberg, M., and Hanenberg, S. What is the impact of static type systems on debugging type errors and semantic errors? An empirical study of differences in debugging time using statically and dynamically typed languages - yet published work.

[28]

Stuchlik, A., and Hanenberg, S. Static vs. dynamic type systems: An empirical study about the relationship between type casts and development time. In Proceedings of the 7th symposium on Dynamic languages (Portland, Oregon, USA, 2011), DLS '11, ACM, pp. 97--106.

Digital Library

[29]

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA, USA, 2000.

Digital Library

Cited By

Troppmann DFass AStaicu CFilkov VRay BZhou M(2024)Typed and Confused: Studying the Unexpected Dangers of Gradual TypingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695549(1858-1870)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695549
Chen ZChen LYang YFeng QLi XSong W(2024)Risky Dynamic Typing-related Practices in Python: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/364959333:6(1-35)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3649593
Kazerouni ALehr JWood ZStephenson BStone JBattestilli LRebelsky SShoop L(2024)Community Action Computing: A Data-centric CS0 CourseProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630807(646-652)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630807
Show More Cited By

Index Terms

An empirical study of the influence of static type systems on the usability of undocumented software
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language features

Recommendations

An experiment about static and dynamic type systems: doubts about the positive impact of static type systems on development time
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applications

Although static type systems are an essential part in teach-ing and research in software engineering and computer science, there is hardly any knowledge about what the impact of static type systems on the development time or the resulting quality for a ...
An experiment about static and dynamic type systems: doubts about the positive impact of static type systems on development time
OOPSLA '10

Although static type systems are an essential part in teach-ing and research in software engineering and computer science, there is hardly any knowledge about what the impact of static type systems on the development time or the resulting quality for a ...
Static vs. dynamic type systems: an empirical study about the relationship between type casts and development time
DLS '11

Static type systems are essential in computer science. However, there is hardly any knowledge about the impact of type systems on the resulting piece of software. While there are authors that state that static types increase the development speed, other ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 47, Issue 10

OOPSLA '12

October 2012

1011 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/2398857

Issue’s Table of Contents

OOPSLA '12: Proceedings of the ACM international conference on Object oriented programming systems languages and applications
October 2012
1052 pages
ISBN:9781450315616
DOI:10.1145/2384616
General Chair:
Gary T. Leavens
University of Central Florida
,
Program Chair:
Matthew B. Dwyer
University of Nebraska - Lincoln

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2012

Published in SIGPLAN Volume 47, Issue 10

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

73
Total Citations
View Citations
984
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Troppmann DFass AStaicu CFilkov VRay BZhou M(2024)Typed and Confused: Studying the Unexpected Dangers of Gradual TypingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695549(1858-1870)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695549
Chen ZChen LYang YFeng QLi XSong W(2024)Risky Dynamic Typing-related Practices in Python: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/364959333:6(1-35)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3649593
Kazerouni ALehr JWood ZStephenson BStone JBattestilli LRebelsky SShoop L(2024)Community Action Computing: A Data-centric CS0 CourseProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630807(646-652)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630807
Xu WChen LSu CGuo YLi YZhou YXu B(2023)How Well Static Type Checkers Work with Gradual Typing? A Case Study on Python2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC)10.1109/ICPC58990.2023.00039(242-253)Online publication date: May-2023
https://doi.org/10.1109/ICPC58990.2023.00039
Jin WZhong DDing ZFan MLiu TGrundy J(2021)Where to startProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678947(529-541)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678947
Tunnell Wilson PGreenman BPombrio JKrishnamurthi S(2020)The behavior of gradual types: a user studyACM SIGPLAN Notices10.1145/3393673.327694753:8(1-12)Online publication date: 6-Apr-2020
https://dl.acm.org/doi/10.1145/3393673.3276947
Tunnell Wilson PGreenman BPombrio JKrishnamurthi SFelgentreff T(2018)The behavior of gradual types: a user studyProceedings of the 14th ACM SIGPLAN International Symposium on Dynamic Languages10.1145/3276945.3276947(1-12)Online publication date: 24-Oct-2018
https://dl.acm.org/doi/10.1145/3276945.3276947
Wainer JXavier E(2018)A Controlled Experiment on Python vs C for an Introductory Programming CourseACM Transactions on Computing Education10.1145/315289418:3(1-16)Online publication date: 9-Aug-2018
https://dl.acm.org/doi/10.1145/3152894
Ray BPosnett DDevanbu PFilkov V(2017)A large-scale study of programming languages and code quality in GitHubCommunications of the ACM10.1145/312690560:10(91-100)Online publication date: 25-Sep-2017
https://dl.acm.org/doi/10.1145/3126905
Faul F(2017)Toward a Perceptually Uniform Parameter Space for Filter TransparencyACM Transactions on Applied Perception10.1145/302273214:2(1-21)Online publication date: 2-Jan-2017
https://dl.acm.org/doi/10.1145/3022732
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents