short-paper

The hidden cost of code completion: understanding the impact of the recommendation-list length on its efficiency

Authors:

Xianhao Jin,

Francisco ServantAuthors Info & Claims

MSR '18: Proceedings of the 15th International Conference on Mining Software Repositories

Pages 70 - 73

https://doi.org/10.1145/3196398.3196474

Published: 28 May 2018 Publication History

Get Access

Abstract

Automatic code completion is a useful and popular technique that software developers use to write code more effectively and efficiently. However, while the benefits of code completion are clear, its cost is yet not well understood. We hypothesize the existence of a hidden cost of code completion, which mostly impacts developers when code completion techniques produce long recommendations. We study this hidden cost of code completion by evaluating how the length of the recommendation list affects other factors that may cause inefficiencies in the process. We study how common long recommendations are, whether they often provide low-ranked correct items, whether they incur longer time to be assessed, and whether they were more prevalent when developers did not select any item in the list. In our study, we observe evidence for all these factors, confirming the existence of a hidden cost of code completion.

References

[1]

Muhammad Asaduzzaman, Chanchal K Roy, Kevin A Schneider, and Daqing Hou. 2014. Cscc: Simple, efficient, context sensitive code completion. In Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on. IEEE, 71--80.

Digital Library

Google Scholar

[2]

Mohammad Ghafari and Hamidreza Moradi. 2017. A framework for classifying and comparing source code recommendation systems. In Software Analysis, Evolution and Reengineering (SANER), 2017 IEEE 24th International Conference on. IEEE, 555--556.

Google Scholar

[3]

Xianhao Jin and Francisco Servant. 2018. The Hidden Cost of Code Completion: Understanding the Impact of the Recommendation-list Length on its Efficiency. (March 2018).

Digital Library

Google Scholar

[4]

Sebastian Proksch, Sven Amann, and Sarah Nadi. 2018. Enriched Event Streams: A General Dataset for Empirical Studies on In-IDE Activities of Software Developers. In Proceedings of the 15th Working Conference on Mining Software Repositories.

Digital Library

Google Scholar

[5]

Sebastian Proksch, Sven Amann, Sarah Nadi, and Mira Mezini. 2016. Evaluating the evaluations of code recommender systems: A reality check. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 111--121.

Digital Library

Google Scholar

[6]

Sebastian Proksch, Johannes Lerch, and Mira Mezini. 2015. Intelligent code completion with Bayesian networks. ACM Transactions on Software Engineering and Methodology (TOSEM) 25, 1 (2015), 3.

Digital Library

Google Scholar

[7]

Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Acm Sigplan Notices, Vol. 49. ACM, 419--428.

Digital Library

Google Scholar

[8]

Romain Robbes and Michele Lanza. 2010. Improving code completion with program history. Automated Software Engineering 17, 2 (2010), 181--212.

Digital Library

Google Scholar

Cited By

View all

Weber TBrandmaier MSchmidt AMayer S(2024)Significant Productivity Gains through Programming with Large Language ModelsProceedings of the ACM on Human-Computer Interaction10.1145/36611458:EICS(1-29)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661145
Aldndni WServant FMeng NBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Understanding the Impact of Branch Edit Features for the Automatic Prediction of Merge Conflict ResolutionsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644433(149-160)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644433
Ciniselli MMartin-Lopez ABavota GBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)On the Generalizability of Deep Learning-based Code Completion Across Programming Language VersionsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644411(99-111)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644411
Show More Cited By

Index Terms

The hidden cost of code completion: understanding the impact of the recommendation-list length on its efficiency
1. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments
    2. Software maintenance tools

Recommendations

Learning from examples to improve code completion systems
ESEC/FSE '09: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering

The suggestions made by current IDE's code completion features are based exclusively on static type system of the programming language. As a result, often proposals are made which are irrelevant for a particular working context. Also, these suggestions ...
Principled syntactic code completion using placeholders
SLE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Software Language Engineering

Principled syntactic code completion enables developers to change source code by inserting code templates, thus increasing developer efficiency and supporting language exploration. However, existing code completion systems are ad-hoc and neither ...
Exploring and Improving Code Completion for Test Code
ICPC '24: Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension

Code completion is an important feature in Integrated Development Environments (IDEs). These years, researchers have been making efforts for intelligent code completion. However, existing work on intelligent code completion either only considered ...

Comments

Information & Contributors

Information

Published In

MSR '18: Proceedings of the 15th International Conference on Mining Software Repositories

May 2018

627 pages

ISBN:9781450357166

DOI:10.1145/3196398

General Chair:
Andy Zaidman
Delft University of Technology, Netherlands
,
Program Chairs:
Yasutaka Kamei
Kyushu University, Japan
,
Emily Hill
Drew University

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 28 - 29, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
178
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)3

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Weber TBrandmaier MSchmidt AMayer S(2024)Significant Productivity Gains through Programming with Large Language ModelsProceedings of the ACM on Human-Computer Interaction10.1145/36611458:EICS(1-29)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661145
Aldndni WServant FMeng NBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Understanding the Impact of Branch Edit Features for the Automatic Prediction of Merge Conflict ResolutionsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644433(149-160)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644433
Ciniselli MMartin-Lopez ABavota GBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)On the Generalizability of Deep Learning-based Code Completion Across Programming Language VersionsProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644411(99-111)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644411
Jin XFeng YWang CLiu YHu YGao YXia KGuo L(2024)PIPELINEASCODE: A CI/CD Workflow Management System through Configuration Files at ByteDance2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00109(1011-1022)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00109
Ciniselli MPascarella LAghajani EScalabrino SOliveto RBavota GGrundy JPollock LPenta M(2023)Source Code Recommender Systems: The Practitioners' PerspectiveProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00182(2161-2172)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00182
Mastropaolo APascarella LGuglielmi ECiniselli MScalabrino SOliveto RBavota GGrundy JPollock LPenta M(2023)On the Robustness of Code Generation Techniques: An Empirical Study on GitHub CopilotProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00181(2149-2160)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00181
Sasano IChoi K(2023)A text-based syntax completion method using LR parsing and its evaluationScience of Computer Programming10.1016/j.scico.2023.102957228:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.scico.2023.102957
Mastropaolo AAghajani EPascarella LBavota G(2023)Automated variable renaming: are we there yet?Empirical Software Engineering10.1007/s10664-022-10274-828:2Online publication date: 14-Feb-2023
https://dl.acm.org/doi/10.1007/s10664-022-10274-8
Izadi MGismondi RGousios GDwyer MDamian DZeller A(2022)CodeFillProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510172(401-412)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510172
Al Safwan KElarnaoty MServant F(2022)Developers’ need for the rationale of code commitsJournal of Systems and Software10.1016/j.jss.2022.111320189:COnline publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111320
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Learning from examples to improve code completion systems

Principled syntactic code completion using placeholders

Exploring and Improving Code Completion for Test Code