research-article

How Practitioners Perceive the Relevance of ESEM Research

Authors:

Jeffrey C. Carver,

Nicholas A. Kraft,

Thomas ZimmermannAuthors Info & Claims

ESEM '16: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 56, Pages 1 - 10

https://doi.org/10.1145/2961111.2962597

Published: 08 September 2016 Publication History

Abstract

Background: The relevance of ESEM research to industry practitioners is key to the long-term health of the conference. Aims: The goal of this work is to understand how ESEM research is perceived within the practitioner community and provide feedback to the ESEM community ensure our research remains relevant. Method: To understand how practitioners perceive ESEM research, we replicated previous work by sending a survey to several hundred industry practitioners at a number of companies around the world. We asked the survey participants to rate the relevance of the research described in 156 ESEM papers published between 2011 and 2015. Results: We received 9,941 ratings by 437 practitioners who labeled ideas as Essential, Worth-while, Unimportant, or Unwise. The results showed that overall, industrial practitioners find the work published in ESEM to be valuable: 67% of all ratings were essential or worthwhile. We found no correlation between citation count and perceived relevance of the papers. Through a qualitative analysis, we also identified a number of research themes on which practitioners would like to see an increased research focus. Conclusions: The work published in ESEM is generally relevant to industrial practitioners. There are a number of topics for which those practitioners would like to see additional research undertaken.

References

[1]

G. Bavota, B. Dit, R. Oliveto, M. D. Penta, D. Poshyvanyk, and A. D. Lucia. An empirical study on the developers' perception of software coupling. In 35th Intl. Conf. Soft. Engg., pages 692--701, 2013.

Digital Library

[2]

A. Begel and T. Zimmermann. Analyze this! 145 questions for data scientists in software engineering. In 36th Intl. Conf. Soft. Engg., pages 12--13, 2014.

Digital Library

[3]

F. Calefato, F. Lanubile, R. Prikladnicki, and J. a. H. S. Pinto. An empirical simulation-based study of real-time speech translation for multilingual global project teams. In 8th Intl. Symp. on Empirical Soft. Engg. and Measurement, pages 56:1--56:9, 2014.

Digital Library

[4]

L. A. Clarke and D. S. Rosenblum. A historical perspective on runtime assertion checking in software development. ACM SIGSOFT Software Engineering Notes, 31(3):25--37, 2006.

Digital Library

[5]

S. Davies and M. Roper. What's in a bug report? In 8th Int. Symp. on Empirical Soft. Engg. and Measurement, pages 26:1--26:10, 2014.

Digital Library

[6]

D. Delgado and A. Martinez. Cost effectiveness of unit testing: A case study in a financial institution. In 7th Intl. Symp. on Empirical Soft. Engg. and Measurement, pages 340--347, 2013.

[7]

W. Emmerich, M. Aoyama, and J. Sventek. The impact of research on the development of middleware technology. ACM Trans. Softw. Eng. Methodol., 17(4), 2007.

Digital Library

[8]

J. Estublier, D. B. Leblang, A. van der Hoek, R. Conradi, G. Clemm, W. F. Tichy, and D. W. Weber. Impact of software engineering research on the practice of software configuration management. ACM Trans. Softw. Eng. Methodol., 14(4):383--430, 2005.

Digital Library

[9]

H. Huijgens, G. Gousios, and A. van Deursen. Pricing via functional size - a case study of a company's portfolio of 77 outsourced projects. In Intl. Symp. on Empirical Soft. Engg. and Measurement, pages 1--10, 2015.

[10]

B. A. Kitchenham and S. L. Peeger. Guide to Advanced Empirical Software Engineering, chapter Personal Opinion Surveys, pages 63--92. Springer London, London, 2008.

[11]

D. Lo, N. Nagappan, and T. Zimmermann. How practitioners perceive the relevance of software engineering research. In 10th Foundations of Soft. Engg. and European Soft. Engg. Conf., pages 415--425, 2015.

Digital Library

[12]

A. Meneely, H. Srinivasan, A. Musa, A. R. Tejeda, M. Mokary, and B. Spates. When a patch goes bad: Exploring the properties of vulnerability-contributing commits. In Intl. Symp. on Empirical Soft. Engg. and Measurement, pages 65--74, 2013.

[13]

A. N. Meyer, T. Fritz, G. C. Murphy, and T. Zimmermann. Software developers' perceptions of productivity. In 22nd Intl Symp. on Foundations of Soft. Engg., pages 19--29, 2014.

Digital Library

[14]

A. T. Misirli, B. Caglayan, A. Bener, and B. Turhan. A retrospective study of software analytics projects: In-depth interviews with practitioners. IEEE Software, 30(5):54--61, 2013.

Digital Library

[15]

S. Nanz, S. West, K. S. D. Silveira, and B. Meyer. Benchmarking usability and performance of multicore languages. In Int. Symp. on Empirical Soft. Engg. and Measurement, pages 183--192, 2013.

[16]

C. D. Nguyen, B. Mendelson, D. Citron, O. Shehory, T. E. J. Vos, and N. Condori-Fernandez. Evaluating the fittest automated testing tools: An industrial case study. In Intl Symp. on Empirical Soft. Engg. and Measurement, pages 332--339, 2013.

[17]

J. Noll, S. Beecham, and D. Seichter. A qualitative study of open source software development: The open emr project. In Intl. Symp. on Empirical Soft. Engg. and Measurement, pages 30--39, 2011.

Digital Library

[18]

L. J. Osterweil, C. Ghezzi, J. Kramer, and A. L. Wolf. Determining the impact of software engineering research on practice. IEEE Computer, 41(3):39--49, 2008.

Digital Library

[19]

F. Palomba, G. Bavota, M. D. Penta, R. Oliveto, and A. D. Lucia. Do they really smell bad? A study on developers' perception of bad code smells. In 30th IEEE Intl. Conf. on Soft. Maint. and Evolution, pages 101--110, 2014.

Digital Library

[20]

M. Piccioni, C. A. Furia, and B. Meyer. An empirical study of api usability. In Intl. Symp. on Empirical Soft. Engg. and Measurement, pages 5--14, 2013.

[21]

H. D. Rombach, M. Ciolkowski, D. R. Jeffery, O. Laitenberger, F. E. McGarry, and F. Shull. Impact of research on practice in the field of inspections, reviews and walkthroughs: learning from successful industrial uses. ACM SIGSOFT Software Engineering Notes, 33(6):26--35, 2008.

Digital Library

[22]

B. G. Ryder and M. L. Soffa. Influences on the design of exception handling ACM SIGSOFT project on the impact of software engineering research on programming language design. ACM SIGSOFT Software Engineering Notes, 28(4):29--35, 2003.

Digital Library

[23]

B. G. Ryder, M. L. Soffa, and M. M. Burnett. The impact of software engineering research on modern programming languages. ACM Trans. Softw. Eng. Methodol., 14(4):431--477, 2005.

Digital Library

[24]

T. Zimmermann. Card-sorting: From text to themes. In Perspectives on Data Science for Software Engineering. Morgan Kaufmann, 2016.

Cited By

Yu XZhang ZNiu FHu XXia XGrundy JFilkov VRay BZhou M(2024)What Makes a High-Quality Training Dataset for Large Language Models: A Practitioners' PerspectiveProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695061(656-668)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695061
Lo D(2024)Requirements Engineering for Trustworthy Human-AI Synergy in Software Engineering 2.02024 IEEE 32nd International Requirements Engineering Conference (RE)10.1109/RE59067.2024.00011(3-4)Online publication date: 24-Jun-2024
https://doi.org/10.1109/RE59067.2024.00011
Cederbladh JEramo RMuttillo VStrandberg P(2024)Experiences and challenges from developing cyber‐physical systems in industry‐academia collaborationSoftware: Practice and Experience10.1002/spe.331254:6(1193-1212)Online publication date: 17-Jan-2024
https://doi.org/10.1002/spe.3312
Show More Cited By

Recommendations

How practitioners perceive the relevance of software engineering research
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

The number of software engineering research papers over the last few years has grown significantly. An important question here is: how relevant is software engineering research to practitioners in the field? To address this question, we conducted a ...
An overview of LCS research from IWLCS 2019 to 2020
GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion

The International Workshop on Learning Classifier Systems (IWLCS) is a yearly workshop at the GECCO conference where new concepts and results all around Learning Classifier Systems (LCSs) are presented and discussed. One recurring part of the workshop ...
How do Practitioners Perceive the Relevance of Requirements Engineering Research?
<italic>Context</italic>: The relevance of Requirements Engineering (RE) research to practitioners is vital for a long-term dissemination of research results to everyday practice. Some authors have speculated about a mismatch between research and practice ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '16: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

September 2016

457 pages

ISBN:9781450344272

DOI:10.1145/2961111

General Chair:
Marcela Genero
University of Castilla-La Mancha, Spain
,
Program Chairs:
Andreas Jedlitschka
Fraunhofer IESE, Germany
,
Magne Jørgensen
Simula Research Laboratory, Norway
,
Giuseppe Scanniello
University of Basilicata, Italy
,
Sreedevi Sampath
University of Maryland Baltimore County, USA
,
Danilo Caivano
SER&Practices, Italy
,
Daniel Port
University of Hawaii, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ESEM '16

Sponsor:

SIGSOFT

ESEM '16: ACM/IEEE 9th International Symposium on Empirical Software Engineering and Measurement

September 8 - 9, 2016

Ciudad Real, Spain

Acceptance Rates

ESEM '16 Paper Acceptance Rate 27 of 122 submissions, 22%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
320
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yu XZhang ZNiu FHu XXia XGrundy JFilkov VRay BZhou M(2024)What Makes a High-Quality Training Dataset for Large Language Models: A Practitioners' PerspectiveProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695061(656-668)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695061
Lo D(2024)Requirements Engineering for Trustworthy Human-AI Synergy in Software Engineering 2.02024 IEEE 32nd International Requirements Engineering Conference (RE)10.1109/RE59067.2024.00011(3-4)Online publication date: 24-Jun-2024
https://doi.org/10.1109/RE59067.2024.00011
Cederbladh JEramo RMuttillo VStrandberg P(2024)Experiences and challenges from developing cyber‐physical systems in industry‐academia collaborationSoftware: Practice and Experience10.1002/spe.331254:6(1193-1212)Online publication date: 17-Jan-2024
https://doi.org/10.1002/spe.3312
Rahman ABose DBarsha FPandita R(2023)Defect Categorization in Compilers: A Multi-vocal Literature ReviewACM Computing Surveys10.1145/362631356:4(1-42)Online publication date: 10-Nov-2023
https://dl.acm.org/doi/10.1145/3626313
Parracho TZacarias RSeruffo Mdos Santos R(2023)I didn’t find what I wanted - How do Developers Consume Information in Software Ecosystems Portals?Proceedings of the XIX Brazilian Symposium on Information Systems10.1145/3592813.3592899(143-150)Online publication date: 29-May-2023
https://dl.acm.org/doi/10.1145/3592813.3592899
Badampudi DUnterkalmsteiner MBritto R(2023)Modern Code Reviews—Survey of Literature and PracticeACM Transactions on Software Engineering and Methodology10.1145/358500432:4(1-61)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3585004
MURAKAMI YTSUNODA M(2022)Replicated Study of Effectiveness Evaluation of Cutting-Edge Software EngineeringIEICE Transactions on Information and Systems10.1587/transinf.2021MPL0002E105.D:1(21-25)Online publication date: 1-Jan-2022
https://doi.org/10.1587/transinf.2021MPL0002
Tuna EKovalenko VTüzün EHarman MMiller H(2022)Bug tracking process smells in practiceProceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice10.1145/3510457.3513080(77-86)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510457.3513080
Franch XMendez DVogelsang AHeldal RKnauss EOriol MTravassos GCarver JZimmermann T(2022)How do Practitioners Perceive the Relevance of Requirements Engineering Research?IEEE Transactions on Software Engineering10.1109/TSE.2020.304274748:6(1947-1964)Online publication date: 1-Jun-2022
https://doi.org/10.1109/TSE.2020.3042747
Tuna EKovalenko VTuzun E(2022)Bug Tracking Process Smells In Practice2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)10.1109/ICSE-SEIP55303.2022.9793952(77-86)Online publication date: May-2022
https://doi.org/10.1109/ICSE-SEIP55303.2022.9793952
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents