research-article

Analysis of Overhead in Dynamic Java Performance Monitoring

Authors:

Vojtěch Horký,

Jaroslav Kotrč,

Petr TůmaAuthors Info & Claims

ICPE '16: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering

Pages 275 - 286

https://doi.org/10.1145/2851553.2851569

Published: 12 March 2016 Publication History

Abstract

In production environments, runtime performance monitoring is often limited to logging of high level events. More detailed measurements, such as method level tracing, tend to be avoided because their overhead can disrupt execution. This limits the information available to developers when solving performance issues at code level. One approach that reduces the measurement disruptions is dynamic performance monitoring, where the measurement instrumentation is inserted and removed as needed. Such selective monitoring naturally reduces the aggregate overhead, but also introduces transient overhead artefacts related to insertion and removal of instrumentation. We experimentally analyze this overhead in Java, focusing in particular on the measurement accuracy, the character of the transient overhead, and the longevity of the overhead artefacts.

Among other results, we show that dynamic monitoring requires time from seconds to minutes to deliver stable measurements, that the instrumentation can both slow down and speed up the execution, and that the overhead artefacts can persist beyond the monitoring period.

References

[1]

AspectJ, 2015. http://eclipse.org/aspectj.

[2]

H. Boehm. Can seqlocks get along with programming language memory models?, 2012. http://www.hpl.hp.com/techreports/2012/HPL-2012--68.html.

Digital Library

[3]

L. Bulej, T. Bureš, J. Keznikl, A. Koubková, A. Podzimek, and P. Tůma. Capturing performance assumptions using stochastic performance logic. In Proc. ICPE 2012, pages 311--322, New York, NY, USA, 2012. ACM.

Digital Library

[4]

B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In Proc. USENIX 2004, 2004.

Digital Library

[5]

A. Diaconescu, A. Mos, and J. Murphy. Automatic performance management in component based software systems. In Proc. ICAC 2004, 2004.

Digital Library

[6]

B. Dufour, C. Goard, L. Hendren, O. de Moor, G. Sittampalam, and C. Verbrugge. Measuring the dynamic behaviour of AspectJ programs. In Proc. OOPSLA 2004, pages 150--169, New York, NY, USA, 2004. ACM.

Digital Library

[7]

J. Ehlers, A. van Hoorn, J. Waller, and W. Hasselbring. Self-adaptive software system monitoring for performance anomaly localization. In Proc. ICAC 2011, pages 197--200, Karlsruhe, Germany, June 2011. ACM.

Digital Library

[8]

H. Eichelberger and K. Schmid. Flexible resource monitoring of Java programs. Journal of Systems and Software, 93:163--186, July 2014.

[9]

K. Govindraj, S. Narayanan, B. Thomas, P. Nair, and S. Peeru. On using AOP for application performance management. In Proc. AOSD 2006, pages 18--30, 2006.

[10]

M. Grimmer, M. Rigger, L. Stadler, R. Schatz, and H. Mössenböck. An efficient native function interface for Java. In Proc. PPPJ 2013, pages 35--44, New York, NY, USA, 2013. ACM.

Digital Library

[11]

M. Haupt and M. Mezini. Micro-measurements for dynamic aspect-oriented systems. In Object-Oriented and Internet-Based Technologies, number 3263 in LNCS, pages 81--96. Springer Berlin Heidelberg, 2004.

[12]

V. Horký, P. Libič, L. Marek, A. Steinhauser, and P. Tůma. Utilizing performance unit tests to increase performance awareness. In Proc. ICPE 2015, pages 289--300, New York, NY, USA, 2015. ACM.

Digital Library

[13]

T. Kalibera, L. Bulej, and P. Tuma. Benchmark precision and random initial state. In Proc. SPECTS 2005, pages 853--862. SCS, 2005.

[14]

T. Kotzmann, C. Wimmer, H. Mössenböck, T. Rodriguez, K. Russell, and D. Cox. Design of the java hotspot client compiler for java 6. ACM Transactions on Architecture and Code Optimization, 5(1):7:1--7:32, May 2008.

Digital Library

[15]

D. Lea. The JSR-133 cookbook for compiler writers, 2011. http://gee.cs.oswego.edu/dl/jmm/cookbook.html.

[16]

P. Libi\v c, L. Bulej, V. Horky, and P. T\ruma. On the limits of modeling generational garbage collector performance. In Proc. ICPE 2014, pages 15--26, New York, NY, USA, 2014. ACM.

Digital Library

[17]

P. Libic, L. Bulej, V. Horký, and P. T\ruma. Estimating the impact of code additions on garbage collection overhead. In Proc. EPEW 2015, number 9272 in LNCS, pages 130--145. Springer International Publishing, Aug. 2015.

[18]

A. D. Malony. Performance Observability. PhD thesis, University of Illinois at Urbana-Champaign, Champaign, IL, USA, 1990. AAI9114332.

Digital Library

[19]

A. D. Malony and S. S. Shende. Overhead compensation in performance profiling. In Proc. Euro-Par 2004, number 3149 in LNCS, pages 119--132. Springer Berlin Heidelberg, Aug. 2004.

[20]

A. D. Malony and S. S. Shende. Models for on-the-fly compensation of measurement overhead in parallel performance profiling. In Proc. Euro-Par 2005, number 3648 in LNCS, pages 72--82. Springer Berlin Heidelberg, Aug. 2005.

Digital Library

[21]

L. Marek, A. Villazón, Y. Zheng, D. Ansaloni, W. Binder, and Z. Qi. DiSL: A domain-specific language for bytecode instrumentation. In Proc. AOSD 2012, pages 239--250, New York, NY, USA, 2012. ACM.

Digital Library

[22]

T. Martinec, L. Marek, A. Steinhauser, P. Tůma, Q. Noorshams, A. Rentschler, and R. Reussner. Constructing performance model of JMS middleware platform. In Proc. ICPE 2014, pages 123--134, New York, NY, USA, 2014. ACM.

Digital Library

[23]

A. Mos and J. Murphy. COMPAS: Adaptive performance monitoring of component-based systems. In Proc. ICSE 2004 RAMSS, 2004.

[24]

Oracle. Java microbenchmark harness, 2013--2015. http://openjdk.java.net/projects/code-tools/jmh.

[25]

P. Panchamukhi. Kernel debugging with kprobes, 2004. http://www.ibm.com/developerworks/library/l-kprobes/index.html.

[26]

N. Park, B. Hong, and V. Prasanna. Tiling, block data layout, and memory hierarchy performance. IEEE Transactions on Parallel and Distributed Systems, 14(7):640--654, July 2003.

Digital Library

[27]

T. Parsons, A. Mos, and J. Murphy. Non-intrusive end-to-end runtime path tracing for J2EE systems. Software, IEEE Proceedings, 153(4):149--161, Aug. 2006.

[28]

D. J. Pearce, M. Webster, R. Berry, and P. H. J. Kelly. Profiling with AspectJ. Software: Practice and Experience, 37(7):747--777, June 2007.

Digital Library

[29]

A. Sarimbekov, A. Sewe, W. Binder, P. Moret, and M. Mezini. JP2: Call-site aware calling context profiling for the Java Virtual Machine. Science of Computer Programming, 79:146--157, Jan. 2014.

Digital Library

[30]

S. S. Shende and A. D. Malony. The Tau parallel performance system. Int. J. High Perform. Comput. Appl., 20(2):287--311, May 2006.

Digital Library

[31]

A. Shipilëv. Java Microbenchmark Harness (The Lesser of Two Evils). Presentation at Devoxx, 2013. http://shipilev.net/talks/devoxx-Nov2013-benchmarking.pdf.

[32]

SPASS-meter monitoring framework, 2015. http://www.sse.uni-hildesheim.de/spass-meter.

[33]

SPEC Java server business benchmark, 2015. http://www.spec.org/jbb2015.

[34]

A. Tamches and B. P. Miller. Fine-grained dynamic instrumentation of commodity operating system kernels. In Proc. OSDI 1999, pages 117--130, Berkeley, CA, USA, 1999. USENIX Association.

Digital Library

[35]

A. van Hoorn, M. Rohr, W. Hasselbring, J. Waller, J. Ehlers, S. Frey, and D. Kieselhorst. Continuous monitoring of software services: Design and application of the Kieker framework. Report, Department of Computer Science, Kiel University, Germany, Nov. 2009.

[36]

A. van Hoorn, J. Waller, and W. Hasselbring. Kieker: A framework for application performance monitoring and dynamic software analysis. In Proc. ICPE 2012, pages 247--248, Boston, Massachusetts, USA, Apr. 2012. ACM.

Digital Library

[37]

J. Waller, F. Fittkau, and W. Hasselbring. Application performance monitoring: Trade-off between overhead reduction and maintainability. In Proceedings of the Symposium on Software Performance 2014, pages 1--24, Stuttgart, Germany, Nov. 2014. University of Stuttgart.

Cited By

Reichelt DBulej LJung Rvan Hoorn ABalsamo SKnottenbelt WAbad CShang W(2024)Overhead Comparison of Instrumentation FrameworksCompanion of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629527.3652269(249-256)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629527.3652269
Baluta ARouf YMukherjee JJiang ZLitoiu MBalsamo SKnottenbelt WAbad CShang W(2024)Disambiguating Performance Anomalies from Workload Changes in Cloud-Native ApplicationsProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645046(286-297)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645046
Reichelt DKühne SHasselbring WVieira MCardellini VDi Marco ATuma P(2023)Towards Solving the Challenge of Minimal Overhead MonitoringCompanion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3584851(381-388)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3584851
Show More Cited By

Index Terms

Analysis of Overhead in Dynamic Java Performance Monitoring
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Performance
2. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Low overhead program monitoring and profiling

Program instrumentation, inserted either before or during execution, is rapidly becoming a necessary component of many systems. Instrumentation is commonly used to collect information for many diverse analysis applications, such as detecting program ...
Low overhead program monitoring and profiling
PASTE '05: Proceedings of the 6th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering

Program instrumentation, inserted either before or during execution, is rapidly becoming a necessary component of many systems. Instrumentation is commonly used to collect information for many diverse analysis applications, such as detecting program ...
Dynamic instrumentation for Java using a virtual JVM
Performance analysis and grid computing

Dynamic instrumentation, meaning modification of an application's instructions at run-time in order to monitor its behaviour, is a very powerful foundation for a wide range of program manipulation tools. This paper concerns the problem of implementing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '16: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering

March 2016

346 pages

ISBN:9781450340809

DOI:10.1145/2851553

General Chairs:
Alberto Avritzer
Sonatype, Inc., USA
,
Alexandru Iosup
Delft University of Technology, the Netherlands
,
Program Chairs:
Xiaoyun Zhu
Futurewei Technologies, USA
,
Steffen Becker
Chemnitz University of Technology, Germany

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 March 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICPE'16

Sponsor:

ICPE'16: ACM/SPEC International Conference on Performance Engineering

March 12 - 16, 2016

Delft, The Netherlands

Acceptance Rates

ICPE '16 Paper Acceptance Rate 23 of 74 submissions, 31%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
188
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Reichelt DBulej LJung Rvan Hoorn ABalsamo SKnottenbelt WAbad CShang W(2024)Overhead Comparison of Instrumentation FrameworksCompanion of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629527.3652269(249-256)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629527.3652269
Baluta ARouf YMukherjee JJiang ZLitoiu MBalsamo SKnottenbelt WAbad CShang W(2024)Disambiguating Performance Anomalies from Workload Changes in Cloud-Native ApplicationsProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645046(286-297)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645046
Reichelt DKühne SHasselbring WVieira MCardellini VDi Marco ATuma P(2023)Towards Solving the Challenge of Minimal Overhead MonitoringCompanion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3584851(381-388)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3584851
Trubiani CPinciroli RBiaggi AFontana F(2023)Automated Detection of Software Performance Antipatterns in Java-Based ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2023.323432149:4(2873-2891)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2023.3234321
Mertz JNunes I(2023)Software runtime monitoring with adaptive sampling rate to collect representative samples of execution tracesJournal of Systems and Software10.1016/j.jss.2023.111708202:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111708
Mukherjee JKrishnamurthy D(2020)PRIMA: Subscriber-Driven Interference Mitigation for Cloud ServicesIEEE Transactions on Network and Service Management10.1109/TNSM.2019.296161317:2(958-971)Online publication date: 10-Jun-2020
https://dl.acm.org/doi/10.1109/TNSM.2019.2961613
Mukherjee JBaluta ALitoiu MKrishnamurthy D(2020)RAD: Detecting Performance Anomalies in Cloud-based Web Services2020 IEEE 13th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD49709.2020.00073(493-501)Online publication date: Oct-2020
https://doi.org/10.1109/CLOUD49709.2020.00073
Mertz JNunes ILitoiu MClarke STei K(2019)On the practical feasibility of software monitoringProceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1109/SEAMS.2019.00030(169-180)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/SEAMS.2019.00030
Gerostathopoulos IBures TSchmid SHorky VPrehofer CTuma PDrira KOquendo FBabar MLegay A(2016)Towards systematic live experimentation in software-intensive systems of systemsProceedings of the International Colloquium on Software-intensive Systems-of-Systems at 10th European Conference on Software Architecture10.1145/3175731.3176175(1-7)Online publication date: 29-Nov-2016
https://dl.acm.org/doi/10.1145/3175731.3176175

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents