research-article

Evaluating the accuracy of Java profilers

Authors:

Todd Mytkowicz,

Matthias Hauswirth,

Peter F. SweeneyAuthors Info & Claims

ACM SIGPLAN Notices, Volume 45, Issue 6

Pages 187 - 197

https://doi.org/10.1145/1809028.1806618

Published: 05 June 2010 Publication History

Abstract

Performance analysts profile their programs to find methods that are worth optimizing: the "hot" methods. This paper shows that four commonly-used Java profilers (xprof, hprof, jprofile, and yourkit) often disagree on the identity of the hot methods. If two profilers disagree, at least one must be incorrect. Thus, there is a good chance that a profiler will mislead a performance analyst into wasting time optimizing a cold method with little or no performance improvement.

This paper uses causality analysis to evaluate profilers and to gain insight into the source of their incorrectness. It shows that these profilers all violate a fundamental requirement for sampling based profilers: to be correct, a sampling-based profilermust collect samples randomly.

We show that a proof-of-concept profiler, which collects samples randomly, does not suffer from the above problems. Specifically, we show, using a number of case studies, that our profiler correctly identifies methods that are important to optimize; in some cases other profilers report that these methods are cold and thus not worth optimizing.

References

[1]

B. Alpern, C. R. Attanasio, J. J. Barton, M. G. Burke, P. Cheng, J.-D. Choi, A. Cocchi, S. J. Fink, D. Grove, M. Hind, S. F. Hummel, D. Lieber, V. Litvinov, M. F. Mergen, T. Ngo, J. R. Russell, V. Sarkar, M. J. Serrano, J. C. Shepherd, S. E. Smith, V. C. Sreedhar, H. Srinivasan, and J. Whaley. The Jalapeño virtual machine. IBM Systems Journal, 39(1):211--238, February 2000.

Digital Library

[2]

M. Arnold and D. Grove. Collecting and exploiting high-accuracy call graph profiles in virtual machines. In Proc. of Int'l Symposium on Code Generation and Optimization, pages 51--62, Los Alamos, CA, March 2005. IEEE Computer Society.

Digital Library

[3]

S. M. Blackburn, R. Garner, C. Hoffman, A. M. Khan, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanović, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In Proc. of ACM SIGPLAN Conf. on Object-Oriented Programing, Systems, Languages, and Applications, pages 169--190, Portland, OR, Oct. 2006. ACM.

Digital Library

[4]

S. M. Blackburn, P. Cheng, and K. S. Mckinley. Myths and realities: The performance impact of garbage collection. In Proc. of ACM SIMETRICS Conf. onMeasurement andModeling Computer Systems, pages 25--36, New York, NY, Jan. 2004. ACM.

Digital Library

[5]

D. Buytaert, A. Georges, M. Hind, M. Arnold, L. Eeckhout, and K. De Bosschere. Using HPM-sampling to drive dynamic compilation. In Proc. of ACM SIGPLAN Conf. on Object-Oriented Programing, Systems, Languages, and Applications, pages 553--568, Montreal, Canada, Oct. 2007. ACM.

Digital Library

[6]

Dehao Chen, Neil Vachharajani, and Robert Hundt. Taming hardware event samples for fdo compilation. International Symposium on Code Generation and Optimization (CGO), 2010.

Digital Library

[7]

A. Diwan, E. Moss, and R. Hudson. Compiler support for garbage collection in a statically typed language. SIGPLAN Not., 27(7):273--282, 1992.

Digital Library

[8]

M. Dmitriev. Selective profiling of Java applications using dynamic bytecode instrumentation. In Proc. of IEEE Int'l Symposium on Performance Analysis of Systems and Software, pages 141--150, Washington, DC, March 2004. IEEE.

Digital Library

[9]

E. Duesterwald and V. Bala. Software profiling for hot path prediction: less is more. SIGPLAN Not., 35(11):202--211, 2000.

Digital Library

[10]

Eclipse: Open source java profiler v4.6.1. http://www.eclipse.org/tptp/.

[11]

Ej technologies: Commercial java profiler. http://www.ejtechnologies.com/products/jprofiler/overview.html.

[12]

A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous Java performance evaluation. In Proc. of ACM SIGPLAN Conf. on Objectoriented Programming, Systems, Languages and Applications, pages 57--76, Montreal, Canada, Oct. 2007. ACM.

Digital Library

[13]

S. L. Graham, P. B. Kessler, and M. K. Mckusick. Gprof: A call graph execution profiler. In Proc. of ACM SIGPLAN Symposium on Compiler Construction, pages 120--126, Boston, Mass., 1982. ACM.

Digital Library

[14]

D. Gu, C. Verbrugge, and E. Gagnon. Code layout as a source of noise in JVM performance. Studia Informatica Universalis, pages 83--99, 2004.

[15]

R. Hegger, H. Kantz, and T. Schreiber. Practical implementation of nonlinear time series methods: The TISEAN package. Chaos, 9(2):413--435, 1999.

[16]

Sam Kash Kachigan. Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods. Radius Press, 1986.

[17]

S. Mccanne and C. Torek. A randomized sampling clock for CPU utilization estimation and code profiling. In Proc. of the Winter USENIX Conf., pages 387--394, San Diego, CA, Jan. 1993.

[18]

T. Moseley, A. Shye, V.J. Reddi, D. Grunwald, and R. Peri. Shadow profiling: Hiding instrumentation costs with parallelism. In Proc. of Int'l Symposium on Code Generation and Optimization, pages 198--208, Washington, DC, March 2007. IEEE Computer Society.

Digital Library

[19]

T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney. Producing wrong data without doing anything obviously wrong! In Proc. of Int'l Conf. on Architectural Support for Programming Languages and Operating Systems, pages 265--276, Washington, DC, March 2009. ACM.

Digital Library

[20]

Netbeans: Open source java profiler. v6.7. http://profiler.netbeans.org/.

[21]

J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 1st edition, 2000.

Digital Library

[22]

S. Rubin, R. Bodík, and T. Chilimbi. An efficient profile-analysis framework for data-layout optimizations. SIGPLAN Not., 37(1):140--153, 2002.

Digital Library

[23]

hprof: an open source java profiler. http://java.sun.com/developer/technicalArticles/Programming/HPROF.html

[24]

xprof: Internal profiler for hotspot. http://java.sun.com/docs/books/performance/1st edition/html/JPAppHotspot.fm.html.

[25]

J. Whaley. A portable sampling-based profiler for java virtual machines. In Proc. of Conf. on Java Grande, pages 78--87, New York, NY, 2000. ACM.

Digital Library

[26]

Yourkit, llc: Commercial java profiler. http://www.yourkit.com

Cited By

Vukasović MProkopec A(2024)Detecting Hot Code from Partially Context-Sensitive Profiles2024 32nd Telecommunications Forum (TELFOR)10.1109/TELFOR63250.2024.10819162(1-8)Online publication date: 26-Nov-2024
https://doi.org/10.1109/TELFOR63250.2024.10819162
Vukasovic MProkopec A(2023)Exploiting Partially Context-sensitive Profiles to Improve Performance of Hot CodeACM Transactions on Programming Languages and Systems10.1145/361293745:4(1-64)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3612937
Trubiani CPinciroli RBiaggi AFontana F(2023)Automated Detection of Software Performance Antipatterns in Java-Based ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2023.323432149:4(2873-2891)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3234321
Show More Cited By

Index Terms

Evaluating the accuracy of Java profilers
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Metrics

Recommendations

Don’t Trust Your Profiler: An Empirical Study on the Precision and Accuracy of Java Profilers
MPLR 2023: Proceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes

To identify optimisation opportunities, Java developers often use sampling profilers that attribute a percentage of run time to the methods of a program. Even so these profilers use sampling, are probabilistic in nature, and may suffer for ...
Evaluating the accuracy of Java profilers
PLDI '10: Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation

Performance analysts profile their programs to find methods that are worth optimizing: the "hot" methods. This paper shows that four commonly-used Java profilers (xprof , hprof , jprofile, and yourkit) often disagree on the identity of the hot methods. ...
Don’t Trust Your Profiler: An Empirical Study on the Precision and Accuracy of Java Profilers (Poster Abstract)
MPLR 2023: Proceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes

To identify optimisation opportunities, Java developers often use sampling profilers that attribute a percentage of run time to the methods of a program. Even so these profilers use sampling, are probabilistic in nature, and may suffer for ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 45, Issue 6

PLDI '10

June 2010

496 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/1809028

Issue’s Table of Contents

PLDI '10: Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation
June 2010
514 pages
ISBN:9781450300193
DOI:10.1145/1806596
General Chair:
Ben Zorn
Microsoft Research
,
Program Chair:
Alex Aiken
Stanford University

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2010

Published in SIGPLAN Volume 45, Issue 6

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

94
Total Citations
View Citations
1,308
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Vukasović MProkopec A(2024)Detecting Hot Code from Partially Context-Sensitive Profiles2024 32nd Telecommunications Forum (TELFOR)10.1109/TELFOR63250.2024.10819162(1-8)Online publication date: 26-Nov-2024
https://doi.org/10.1109/TELFOR63250.2024.10819162
Vukasovic MProkopec A(2023)Exploiting Partially Context-sensitive Profiles to Improve Performance of Hot CodeACM Transactions on Programming Languages and Systems10.1145/361293745:4(1-64)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3612937
Trubiani CPinciroli RBiaggi AFontana F(2023)Automated Detection of Software Performance Antipatterns in Java-Based ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2023.323432149:4(2873-2891)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3234321
Moseler OKreber LDiehl S(2022)The ThreadRadar visualization for debugging concurrent Java programsJournal of Visualization10.1007/s12650-022-00843-w25:6(1267-1289)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s12650-022-00843-w
Weber MApel SSiegmund N(2021)White-Box Performance-Influence ModelsProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00099(1059-1071)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00099
Rosà ARosales EBinder W(2019)Analysis and Optimization of Task Granularity on the Java Virtual MachineACM Transactions on Programming Languages and Systems10.1145/333849741:3(1-47)Online publication date: 16-Jul-2019
https://dl.acm.org/doi/10.1145/3338497
Xu HWang QSong SJohn LLiu XEigenmann RDing CMcKee S(2019)Can we trust profiling results?Proceedings of the ACM International Conference on Supercomputing10.1145/3330345.3330371(284-295)Online publication date: 26-Jun-2019
https://dl.acm.org/doi/10.1145/3330345.3330371
Wang FShi XLu XLima SRocha Á(2018)A low-overhead and efficient Java object profiler on ART virtual machineJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-16965535:3(2997-3009)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.3233/JIFS-169655
Jiang MWang CLuo XMiu MChen T(2017)Characterizing the Impacts of Application Layer DDoS Attacks2017 IEEE International Conference on Web Services (ICWS)10.1109/ICWS.2017.58(500-507)Online publication date: Jun-2017
https://doi.org/10.1109/ICWS.2017.58
St-Amour VAndersen LFelleisen M(2015)Feature-Specific ProfilingCompiler Construction10.1007/978-3-662-46663-6_3(49-68)Online publication date: 2015
https://doi.org/10.1007/978-3-662-46663-6_3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents