A Study of Java Microbenchmark Tail Latencies
Authors: 
Sen He, In Kee Kim, Wei WangAuthors Info & Claims
Sen He, In Kee Kim, Wei WangAuthors Info & ClaimsPages 77 - 81
Abstract
As Java microbenchmarks are great at profiling the performance of essential code elements, they are widely adopted for Java performance testing. Performance testing using Java microbenchmarks is composed of two phases: the warmup phase and the steady phase. Usually, testing results from the warmup phase are discarded because of the highly fluctuating performance caused by the optimization of Java Virtual Machine. The performance results collected during the steady phase are used for performance evaluation as they are assumed to be more stable. However, according to our study, severe performance fluctuations also occur during the steady phase, which leads to long tail latencies. Long tail latencies constitute a big problem in modern Java systems (of course, other systems and applications) as they hurt the user experience by prolonging the overall execution time.
In this paper, we extensively evaluated the long tail performance of 586 Java microbenchmarks from 30 Java systems. The evaluation results show that, for 38% of the benchmarks in steady phase, their 99%tile execution times are over 30% higher than their median execution times. In the worst-case scenario, the 99%ile performance is 659 times higher than the median performance. Furthermore, the 95%ile execution times are above 30% higher than the median execution times for 11% of the steady phase benchmarks.
References
[1]
Edd Barrett, Carl Friedrich Bolz-Tereick, Rebecca Killick, Sarah Mount, and Laurence Tratt. 2017. Virtual Machine Warmup Blows Hot and Cold. Proc. ACM Program. Lang., Vol. 1, OOPSLA, Article 52 (oct 2017), 27 pages. https://doi.org/10.1145/3133876
[2]
Andreas Burger, Heiko Koziolek, Julius Rückert, Marie Platenius-Mohr, and Gösta Stomberg. 2019. Bottleneck Identification and Performance Modeling of OPC UA Communication Models. In Proc. of ACM/SPEC Int'l Conf. on Performance Engineering (ICPE '19).
[3]
Wenzhi Cui, Daniel Richins, Yuhao Zhu, and Vijay Janapa Reddi. April 14, 2019. Tail latency in node.js: energy efficient turbo boosting for long latency requests in event-driven web services. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE'19). Providence, RI, USA.
[4]
Jeffrey Dean and Luiz André Barroso. 2013. The Tail at Scale. Commun. ACM, Vol. 56, 2 (feb 2013), 74--80. https://doi.org/10.1145/2408776.2408794
[5]
Written by Douglas Eadline. [n.d.]. Douglas Eadline. https://www.clustermonkey.net/Benchmarking-Methods/micro-benchmarks-vs-macro-benchmarks.html
[6]
Andy Georges, Dries Buytaert, and Lieven Eeckhout. 2007. Statistically Rigorous Java Performance Evaluation. SIGPLAN Not., Vol. 42, 10 (oct 2007), 57--76. https://doi.org/10.1145/1297105.1297033
[7]
Hao He, Runzhi He, Haiqiao Gu, and Minghui Zhou. August 23--28, 2021a. A large-scale empirical study on Java library migrations: prevalence, trends, and rationales. In 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'21). Athens, Greece.
[8]
Sen He, Tianyi Liu, Palden Lama, Jaewoo Lee, In Kee Kim, and Wei Wang. 2021b. Performance Testing for Cloud Computing with Dependent Data Bootstrapping. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 666--678. https://doi.org/10.1109/ASE51524.2021.9678687
[9]
Sen He, Glenna Manns, John Saunders, Wei Wang, Lori Pollock, and Mary Lou Soffa. 2019. A Statistics-Based Performance Testing Methodology for Cloud Applications. In Proc. of ACM Joint Meeting on European Software Engineering Conf. and Symp. on the Foundations of Software Engineering.
[10]
Zhen Ming Jiang and Ahmed E. Hassan. 2015. A Survey on Load Testing of Large-Scale Software Systems. IEEE Transactions on Software Engineering, Vol. 41, 11 (2015), 1091--1118. https://doi.org/10.1109/TSE.2015.2445340
[11]
Tomas Kalibera and Richard Jones. 2013. Rigorous Benchmarking in Reasonable Time. In Proceedings of the 2013 International Symposium on Memory Management (Seattle, Washington, USA) (ISMM '13). Association for Computing Machinery, New York, NY, USA, 63--74. https://doi.org/10.1145/2491894.2464160
[12]
Christoph Laaber and Philipp Leitner. 2018. An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR '18). Association for Computing Machinery, New York, NY, USA, 119--130. https://doi.org/10.1145/3196398.3196407
[13]
Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner. 2020. Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 989--1001. https://doi.org/10.1145/3368089.3409683
[14]
Philipp Leitner and Cor-Paul Bezemer. 2017. An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (L'Aquila, Italy) (ICPE '17). Association for Computing Machinery, New York, NY, USA, 373--384. https://doi.org/10.1145/3030207.3030213
[15]
Waleed Reda, Marco Canini, P. Lalith Suresh, Dejan Kostic, and Sean Braithwaite. April 23--26, 2017. Rein: Taming Tail Latency in Key-Value Stores via Multiget Scheduling. In ACM European Conference on Computer Systems (EuroSys'17). Belgrade, Serbia.
[16]
Pasindu Tennage, Srinath Perera, Malith Jayasinghe, and Sanath Jayasena. 2019. An Analysis of Holistic Tail Latency Behaviors of Java Microservices. In 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). 697--705. https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00104
[17]
Luca Traini, Vittorio Cortellessa, Daniele Di Pompeo, and Michele Tucci. 2023. Towards Effective Assessment of Steady State Performance in Java Software: Are We There Yet? Empirical Softw. Engg., Vol. 28, 1 (jan 2023), 57 pages. https://doi.org/10.1007/s10664-022--10247-x
[18]
Yutong Zhao, Lu Xiao, Xiao Wang, Bihuan Chen, and Yang Liu. 2019. Localized or Architectural: An Empirical Study of Performance Issues Dichotomy. In Int'l Conf. on Software Engineering: Companion Proceedings.
Index Terms
- A Study of Java Microbenchmark Tail Latencies
Recommendations
Evaluating the Java Native Interface JNI: Leveraging Existing Native Code, Libraries and Threads to a Running Java Virtual Machine
This article aims to explore JNI features and to discover fundamental operations of the Java programming language, such as arrays, objects, classes, threads and exception handling, and to illustrate these by using various algorithms and code samples. ...
A study of Java's non-Java memory
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applicationsA Java application sometimes raises an out-of-memory ex-ception. This is usually because it has exhausted the Java heap. However, a Java application can raise an out-of-memory exception when it exhausts the memory used by Java that is not in the Java ...
A study of Java's non-Java memory
OOPSLA '10A Java application sometimes raises an out-of-memory ex-ception. This is usually because it has exhausted the Java heap. However, a Java application can raise an out-of-memory exception when it exhausts the memory used by Java that is not in the Java ...
Comments
Information & Contributors
Information
Published In
April 2023
421 pages
ISBN:9798400700729
DOI:10.1145/3578245
- General Chairs:
- Marco Vieira,
- Valeria Cardellini,
- Program Chairs:
- Antinisca Di Marco,
- Petr Tuma
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 April 2023
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
ICPE '23
Sponsor:
ICPE '23: ACM/SPEC International Conference on Performance Engineering
April 15 - 19, 2023
Coimbra, Portugal
Acceptance Rates
Overall Acceptance Rate 252 of 851 submissions, 30%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)34
- Downloads (Last 6 weeks)2
Reflects downloads up to 16 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in