short-paper

How is Performance Addressed in DevOps?

Authors:

Cor-Paul Bezemer,

Vincenzo Ferme,

Johannes Grohmann,

Robert Heinrich,

Pooyan Jamshidi,

André van Hoorn,

Monica Villavicencio,

Jürgen Walter,

Felix WillneckerAuthors Info & Claims

ICPE '19: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

Pages 45 - 50

https://doi.org/10.1145/3297663.3309672

Published: 04 April 2019 Publication History

Abstract

DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities --- such as load testing --- require a considerable amount of time to get statistically significant results.

We conducted an industrial survey to get insights into how performance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being used, the granularity of the obtained performance data, and the use of model-based techniques. The survey responses, which come from a wide variety of participants from different industry sectors, indicate that the complexity of performance engineering approaches and tools is a barrier for wide-spread adoption of performance analysis in DevOps. The implication of our results is that performance analysis tools need to have a short learning curve, and should be easy to integrate into the DevOps pipeline in order to be adopted by practitioners.

References

[1]

{n. d.}. The 2017 DevOps Pulse. https://logz.io/wp-content/uploads/2017/07/devops_pulse_2017_final.pdf. ({n. d.}). Accessed: 2018-06-01.

[2]

{n. d.}. 2017 State of DevOps Report. https://www.ipexpoeurope.com/content/download/10069/143970/file/2017-state-of-devops-report.pdf. ({n. d.}). Accessed:2018-06-01.

[3]

{n. d.}.DevOps: The Worst-Kept Secret to Winning in the Applica-tion Economy. https://www.ca.com/content/dam/ca/us/files/white-paper/devops-winning-in-application-economy-2.pdf. ({n. d.}). Accessed: 2018-06-01.

[4]

{n. d.}.KMS Technology Survey: DevOps Enjoying its Moment in the Sun. https://www.kms-technology.com/press/kms-technology-survey-devops-enjoying-its-moment-in-the-sun.({n.d.}). Accessed: 2018-06-01.

[5]

{n. d.}.SPEC RG DevOps Survey - Supplementary Material. https://spec-rgdevops.github.io/specrg_devops_survey/. ({n. d.}). Accessed: 2018-06-01.

[6]

{n. d.}. State of Performance Engineering. https://techbeacon.com/sites/default/files/gated_asset/state-of-performance-engineering-2015-16_final2.pdf. ({n. d.}). Accessed: 2018-06-01.

[7]

{n. d.}. xMatters Atlassian DevOps Maturity. http://info.xmatters.com/rs/178-CPU-592/images/atlassian_devops_survey.pdf ?_ga=2.20537722.378918857.1526304278-746614242.1525703839. ({n. d.}). Accessed: 2018-06-01.

[8]

André B. Bondi. 2014. Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice. Addison-Wesley Professional.

Digital Library

[9]

Andreas Brunnert, Andre van Hoorn, Felix Willnecker, Alexandru Danciu, Wilhelm Hasselbring, Christoph Heger, Nikolas Herbst, Pooyan Jamshidi, Reiner Jung, Joakim von Kistowski, Anne Koziolek, Johannes Kroß, Simon Spinner, Christian Vögele, Jürgen Walter, and Alexander Wert. 2015. Performance-oriented DevOps: A Research Agenda. Technical Report SPEC-RG-2015-01. SPEC Research Group -- DevOps Performance Working Group, Standard Performance Evaluation Corporation (SPEC).

[10]

Alvin C Burns, Ronald F Bush, and Judith Nash. 2008. Basic marketing research: using Microsoft Excel data analysis. Pearson Prentice Hall.

Digital Library

[11]

Vittorio Cortellessa, Antinisca Di Marco, and Paola Inverardi. 2011. Model-Based Software Performance Analysis 1st ed.). Springer Publishing Company, Incorporated.

Digital Library

[12]

F. M. A. Erich, C. Amrit, and M. Daneva. 2017. A Qualitative Study of DevOps Usage in Practice. J. Softw. Evol. Process, Vol. 29, 6 (June 2017), n/a--n/a.

Digital Library

[13]

Arlene Fink. 2012. How to conduct surveys: A step-by-step guide: A step-by-step guide. Sage Publications.

[14]

Christoph Heger, André van Hoorn, Mario Mann, and Duvsan Okanović. 2017. Application Performance Management: State of the Art and Challenges for the Future. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE '17). ACM, 429--432.

Digital Library

[15]

Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). IEEE, 426--437.

Digital Library

[16]

Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation .Addison-Wesley Professional.

Digital Library

[17]

Raj Jain. 1991. The Art of Computer Systems Performance Analysis. John Wiley & Sons.

[18]

Z. M. Jiang and A. E. Hassan. 2015. A Survey on Load Testing of Large-Scale Software Systems. IEEE Transactions on Software Engineering, Vol. 41, 11 (2015), 1091--1118.

Digital Library

[19]

Jonathan Kaldor, Jonathan Mace, Michał Bejda, Edison Gao, Wiktor Kuropatwa, Joe O'Neill, Kian Win Ong, Bill Schaller, Pingjia Shan, Brendan Viscomi, et almbox. 2017. Canopy: An End-to-End Performance Tracing And Analysis System. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP 2017). ACM, 34--50.

Digital Library

[20]

Philipp Leitner and Cor-Paul Bezemer. 2017. An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE). ACM, 373--384.

Digital Library

[21]

Johan Linåker, Sardar Muhammad Sulaman, Rafael Maiani de Mello, and Martin Höst. 2015. Guidelines for Conducting Surveys in Software Engineering.

[22]

Urvs a Reja, Katja Lozar Manfreda, Valentina Hlebec, and Vasja Vehovar. 2003. Open-ended vs. close-ended questions in web questionnaires. Developments in Applied Statistics (Metodolovs ki zvezki), Vol. 19 (2003), 159--77.

[23]

Ralf H Reussner, Steffen Becker, Jens Happe, Robert Heinrich, Anne Koziolek, Heiko Koziolek, Max Kramer, and Klaus Krogmann. 2016. Modeling and simulating software architectures: The Palladio approach. MIT Press.

Digital Library

[24]

Priscilla Salant, I Dillman, and A Don. 1994. How to conduct your own survey. Number 300.723 S3.

[25]

Andre van Hoorn, Pooyan Jamshidi, Philipp Leitner, and Ingo Weber (Eds.). 2017. Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World.

[26]

Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (FSE 2015). ACM, 805--816.

Digital Library

[27]

Liming Zhu, Len Bass, and George Champlin-Scharff. 2016. DevOps and its practices. IEEE Software, Vol. 33, 3 (2016), 32--34.

Cited By

Azad NHyrynsalmi S(2024)Multivocal Literature Review on DevOps Critical Success FactorsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661236(520-527)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661236
Xia YLiao LChen JLi HShang W(2024)Reducing the Length of Field-Replay Based Load TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.340807950:8(1967-1983)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3408079
Jayakody JWijayanayake W(2024)DevOps Maturity; A Systematic Literature Review2024 International Research Conference on Smart Computing and Systems Engineering (SCSE)10.1109/SCSE61872.2024.10550493(1-6)Online publication date: 4-Apr-2024
https://doi.org/10.1109/SCSE61872.2024.10550493
Show More Cited By

Index Terms

How is Performance Addressed in DevOps?
1. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Adoption Issues in DevOps from the Perspective of Continuous Delivery Pipeline
ICSCA '19: Proceedings of the 2019 8th International Conference on Software and Computer Applications

DevOps and Continuous Delivery (CD) are the terms that are always related to each other in Software Delivery and Operation Process area. DevOps introduces a significant agile perspective to deliver the software product in short cycle time that will ...
Understanding devops education with grounded theory
ICSE-SEET '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering Education and Training

DevOps stands for Development-Operations. It arises from the IT industry as a movement aligning development and operations teams. DevOps is broadly recognized as an IT standard, and there is high demand for DevOps practitioners in industry. Since ACM & ...
Understanding DevOps education with Grounded theory
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings

DevOps stands for Development-Operations. It arises from the IT industry as a movement aligning development and operations teams. DevOps is broadly recognized as an IT standard, and there is high demand for DevOps practitioners in industry. Therefore, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '19: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

April 2019

348 pages

ISBN:9781450362399

DOI:10.1145/3297663

General Chairs:
Varsha Apte
IIT Bombay, India
,
Antinisca Di Marco
University of L'Aquila, Italy
,
Program Chairs:
Marin Litoiu
York University, Canada
,
José Merseguer
Universidad de Zaragoza, Spain

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Swiss National Science Foundation project
German Research Foundation (DFG)
German Federal Ministry of Education and Research

Conference

ICPE '19

Sponsor:

ICPE '19: Tenth ACM/SPEC International Conference on Performance Engineering

April 7 - 11, 2019

Mumbai, India

Acceptance Rates

ICPE '19 Paper Acceptance Rate 13 of 71 submissions, 18%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
1,137
Total Downloads

Downloads (Last 12 months)133
Downloads (Last 6 weeks)15

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Azad NHyrynsalmi S(2024)Multivocal Literature Review on DevOps Critical Success FactorsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661236(520-527)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661236
Xia YLiao LChen JLi HShang W(2024)Reducing the Length of Field-Replay Based Load TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.340807950:8(1967-1983)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3408079
Jayakody JWijayanayake W(2024)DevOps Maturity; A Systematic Literature Review2024 International Research Conference on Smart Computing and Systems Engineering (SCSE)10.1109/SCSE61872.2024.10550493(1-6)Online publication date: 4-Apr-2024
https://doi.org/10.1109/SCSE61872.2024.10550493
Eramo RTucci MDi Pompeo DCortellessa VDi Marco ATaibi D(2024)Architectural support for software performance in continuous software engineeringJournal of Systems and Software10.1016/j.jss.2023.111833207:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111833
Azad N(2024)DevOps Challenges and Risk Mitigation Strategies by DevOps Professionals TeamsSoftware Business10.1007/978-3-031-53227-6_26(369-385)Online publication date: 9-Feb-2024
https://doi.org/10.1007/978-3-031-53227-6_26
Straesser MEismann Svon Kistowski JBauer AKounev SVieira MCardellini VDi Marco ATuma P(2023)Autoscaler Evaluation and Configuration: A Practitioner's GuidelineProceedings of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578244.3583721(31-41)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578244.3583721
Jangali MTang YAlexandersson NLeitner PYang JShang W(2023)Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit TestsIEEE Transactions on Software Engineering10.1109/TSE.2022.318800549:4(1704-1725)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3188005
Garbi GIncerto ETribastone M(2023)μP: A Development Framework for Predicting Performance of Microservices by Design2023 IEEE 16th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD60044.2023.00029(178-188)Online publication date: Jul-2023
https://doi.org/10.1109/CLOUD60044.2023.00029
Azad NHyrynsalmi S(2023)DevOps critical success factors — A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2023.107150157:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.infsof.2023.107150
Azad NHyrynsalmi SMäntymäki M(2023)Understanding DevOps Critical Success Factors: Insights from ProfessionalsNew Sustainable Horizons in Artificial Intelligence and Digital Solutions10.1007/978-3-031-50040-4_7(78-90)Online publication date: 15-Dec-2023
https://doi.org/10.1007/978-3-031-50040-4_7
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents