research-article

A Comprehensive Java Benchmark Study on Memory and Garbage Collection Behavior of DaCapo, DaCapo Scala, and SPECjvm2008

Authors:

Philipp Lengauer,

Verena Bitto,

Hanspeter Mössenböck,

Markus WeningerAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 3 - 14

https://doi.org/10.1145/3030207.3030211

Published: 17 April 2017 Publication History

Get Access

Abstract

Benchmark suites are an indispensable part of scientific research to compare different approaches against each another. The diversity of benchmarks is an important asset to evaluate novel approaches for effectiveness and weaknesses. In this paper, we describe the memory characteristics and the GC behavior of commonly used Java benchmarks, i.e., the DaCapo benchmark suite, the DaCapo Scala benchmark suite and the SPECjvm2008 benchmark suite. The paper can serve as a useful guide to select benchmarks in accordance with desired application characteristics on modern virtual machines as well as with different compilers and garbage collectors. It also helps to put results that are based on these benchmarks into perspective. Additionally, we compare Java's current default collector to the G1 GC.

References

[1]

V. Bitto and P. Lengauer. Building custom, efficient, and accurate memory monitoring tools for java applications. In Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering, ICPE '16, pages 321--324, New York, NY, USA, 2016. ACM.

Digital Library

Google Scholar

[2]

S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanović, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. In Proc. of the Annual ACM SIGPLAN Conf. on Object-oriented Programming Systems, Languages, and Applications, pages 169--190, 2006.

Digital Library

Google Scholar

[3]

S. Dieckmann and U. Hoelzle. A study of the allocation behavior of the specjvm98 java benchmarks. Technical report, Santa Barbara, CA, USA, 1998.

Google Scholar

[4]

A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous java performance evaluation. In Proceedings of the 22Nd Annual ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications, OOPSLA '07, pages 57--76, New York, NY, USA, 2007. ACM.

Digital Library

Google Scholar

[5]

A. Georges, L. Eeckhout, and D. Buytaert. Java performance evaluation through rigorous replay compilation. In Proceedings of the 23rd ACM SIGPLAN Conference on Object-oriented Programming Systems Languages and Applications, OOPSLA '08, pages 367--384, New York, NY, USA, 2008. ACM.

Digital Library

Google Scholar

[6]

P. Lengauer, V. Bitto, and H. Mössenböck. Accurate and efficient object tracing for java applications. In Proc. of the 6th ACM/SPEC Int'l. Conf. on Performance Engineering, ICPE '15, pages 51--62, 2015.

Digital Library

Google Scholar

[7]

A. Sewe, M. Mezini, A. Sarimbekov, and W. Binder. Da capo con scala: Design and analysis of a scala benchmark suite for the java virtual machine. In Proceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA '11, pages 657--676, 2011.

Digital Library

Google Scholar

[8]

K. Shiv, K. Chow, Y. Wang, and D. Petrochenko. Specjvm2008 performance characterization. In Proceedings of the 2009 SPEC Benchmark Workshop on Computer Performance Evaluation and Benchmarking, pages 17--35, Berlin, Heidelberg, 2009. Springer-Verlag.

Digital Library

Google Scholar

Cited By

View all

Löff JSchiavio FRosà ABasso MBinder WBalsamo SKnottenbelt WAbad CShang W(2024)Vectorized Intrinsics Can Be Replaced with Pure Java Code without Impairing Steady-State PerformanceProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645051(14-24)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645051
Papadakis OAndronikakis AFoutris NPapadimitriou MStratikopoulos AZakkak FXekalakis PKotselidis CBruno RMoss E(2023)A Multifaceted Memory Analysis of Java BenchmarksProceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes10.1145/3617651.3622978(70-84)Online publication date: 19-Oct-2023
https://dl.acm.org/doi/10.1145/3617651.3622978
Papadakis OAndronikakis AFoutris NPapadimitriou MStratikopoulos AZakkak FXekalakis PKotselidis CBlackburn SPetrank E(2023)Scaling Up Performance of Managed Applications on NUMA SystemsProceedings of the 2023 ACM SIGPLAN International Symposium on Memory Management10.1145/3591195.3595270(1-14)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591195.3595270
Show More Cited By

Index Terms

A Comprehensive Java Benchmark Study on Memory and Garbage Collection Behavior of DaCapo, DaCapo Scala, and SPECjvm2008
1. General and reference
  1. Cross-computing tools and techniques
    1. Evaluation
    2. Performance

Recommendations

The DaCapo benchmarks: java benchmarking development and analysis
OOPSLA '06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications

Since benchmarks drive computer science research and industry product development, which ones we use and how we evaluate them are key questions for the community. Despite complex runtime tradeoffs due to dynamic compilation and garbage collection ...
The DaCapo benchmarks: java benchmarking development and analysis
Proceedings of the 2006 OOPSLA Conference

Since benchmarks drive computer science research and industry product development, which ones we use and how we evaluate them are key questions for the community. Despite complex runtime tradeoffs due to dynamic compilation and garbage collection ...
Controlling garbage collection and heap growth to reduce the execution time of Java applications

In systems that support garbage collection, a tension exists between collecting garbage too frequently and not collecting it frequently enough. Garbage collection that occurs too frequently may introduce unnecessary overheads at the risk of not ...

Comments

Information & Contributors

Information

Published In

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Christian Doppler Forschungsgesellschaft

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
519
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)10

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Löff JSchiavio FRosà ABasso MBinder WBalsamo SKnottenbelt WAbad CShang W(2024)Vectorized Intrinsics Can Be Replaced with Pure Java Code without Impairing Steady-State PerformanceProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645051(14-24)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645051
Papadakis OAndronikakis AFoutris NPapadimitriou MStratikopoulos AZakkak FXekalakis PKotselidis CBruno RMoss E(2023)A Multifaceted Memory Analysis of Java BenchmarksProceedings of the 20th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes10.1145/3617651.3622978(70-84)Online publication date: 19-Oct-2023
https://dl.acm.org/doi/10.1145/3617651.3622978
Papadakis OAndronikakis AFoutris NPapadimitriou MStratikopoulos AZakkak FXekalakis PKotselidis CBlackburn SPetrank E(2023)Scaling Up Performance of Managed Applications on NUMA SystemsProceedings of the 2023 ACM SIGPLAN International Symposium on Memory Management10.1145/3591195.3595270(1-14)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591195.3595270
Tavakolisomeh SBruno RFerreira P(2023)BestGC: An Automatic GC SelectorIEEE Access10.1109/ACCESS.2023.329439811(72357-72373)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3294398
Lambert JMonahan RCasey K(2022)Accidental Choices—How JVM Choice and Associated Build Tools Affect Interpreter PerformanceComputers10.3390/computers1106009611:6(96)Online publication date: 14-Jun-2022
https://doi.org/10.3390/computers11060096
Ye CXu YShen XJin HLiao XSolihin Y(2022)Preserving Addressability Upon GC-Triggered Data Movements on Non-Volatile MemoryACM Transactions on Architecture and Code Optimization10.1145/351170619:2(1-26)Online publication date: 24-Mar-2022
https://dl.acm.org/doi/10.1145/3511706
Ournani ZBelgaid MRouvoy RRust PPenhoat JLanubile F(2021)Evaluating the Impact of Java Virtual Machines on Energy ConsumptionProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3475774(1-11)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3475774
Weninger MGander EMössenböck H(2021)Guided ExplorationProceedings of the ACM on Human-Computer Interaction10.1145/34617315:EICS(1-34)Online publication date: 29-May-2021
https://dl.acm.org/doi/10.1145/3461731
Abhaay SShashank MShreyas BKalambur S(2021)Effect of Garbage Collection on Streaming Big Data Workloads2021 2nd Global Conference for Advancement in Technology (GCAT)10.1109/GCAT52182.2021.9587808(1-6)Online publication date: 1-Oct-2021
https://doi.org/10.1109/GCAT52182.2021.9587808
Nair ASriram ASimon AKalambur SSitaram D(2021)A Comparative Analysis of Garbage Collectors and Their Suitability for Big Data WorkloadsAdvances in Computing and Network Communications10.1007/978-981-33-6977-1_24(305-316)Online publication date: 21-Apr-2021
https://doi.org/10.1007/978-981-33-6977-1_24
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

The DaCapo benchmarks: java benchmarking development and analysis

The DaCapo benchmarks: java benchmarking development and analysis

Controlling garbage collection and heap growth to reduce the execution time of Java applications