research-article

Da capo con scala: design and analysis of a scala benchmark suite for the java virtual machine

Authors:

Aibek Sarimbekov,

Walter BinderAuthors Info & Claims

ACM SIGPLAN Notices, Volume 46, Issue 10

Pages 657 - 676

https://doi.org/10.1145/2076021.2048118

Published: 22 October 2011 Publication History

Abstract

Originally conceived as the target platform for Java alone, the Java Virtual Machine (JVM) has since been targeted by other languages, one of which is Scala. This trend, however, is not yet reflected by the benchmark suites commonly used in JVM research. In this paper, we thus present the design and analysis of the first full-fledged benchmark suite for Scala. We furthermore compare the benchmarks contained therein with those from the well-known DaCapo 9.12 benchmark suite and show where the differences are between Scala and Java code---and where not.

References

[1]

B. Alpern, A. Cocchi, S. Fink, and D. Grove. Efficient implementation of Java interfaces: invoke interface considered harmless. In Proceedings of the 16th Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2001.

Digital Library

[2]

G. Ammons, T. Ball, and J. R. Larus. Exploiting hardware performance counters with flow and context sensitive profiling. In Proceedings of the 10th Conference on Programming Language Design and Implementation (PLDI), 1997.

Digital Library

[3]

M. Bebenita, M. Chang, G. Wagner, A. Gal, C. Wimmer, and M. Franz. Trace-based compilation in execution environments without interpreters. In Proceedings of the 8th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2010.

Digital Library

[4]

W. Binder. A portable and customizable profiling framework for Java based on bytecode instruction counting. In Proceedings of the 3rd Asian Symposium on Programming Languages and Systems (APLAS), 2005.

Digital Library

[5]

W. Binder, J. Hulaas, and P. Moret. A quantitative evaluation of the contribution of native code to Java workloads. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC), 2006.

[6]

W. Binder, J. Hulaas, P. Moret, and A. Villazo ?n. Platform-independent profiling in a virtual execution environment. Software: Practice and Experience, 39(1):47--79, 2009.

Digital Library

[7]

S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J.E.B.Moss,B.Moss,A.Phansalkar,D.Stefanovic, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In Proceedings of the 21st Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2006.

Digital Library

[8]

E. Bodden, A. Sewe, J. Sinschek, M. Mezini, and H. Oueslati. Taming reflection: Aiding static analysis in the presence of reflection and custom class loaders. In Proceedings of the 33rd International Conference on Software Engineering (ICSE), 2011.

Digital Library

[9]

M. Braux and J. Noye. Towards partially evaluating reflection in Java. In Proceedings of the Workshop on Partial Evaluation and Semantics-Based Program Manipulation (PEPM), 1999.

Digital Library

[10]

J. M. Bull, L. A. Smith, M. D. Westhead, D. S. Henty, and R. A. Davey. A methodology for benchmarking Java Grande applications. In Proceedings of the ACM 1999 Conference on Java Grande, 1999.

Digital Library

[11]

Y. Chiba. Redundant boxing elimination by a dynamic compiler for Java. In Proceedings of the 5th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2007.

Digital Library

[12]

S. R. Chidamber and C. F. Kemerer. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20(6):476--493, June 1994. ISSN 0098-5589.

Digital Library

[13]

C. Collberg, G. Myles, and M. Stepp. An empirical study of Java bytecode programs. Software: Practice and Experience, 37(6):581--641, May 2007. ISSN 0038-0644.

Digital Library

[14]

C. Daly, J. Horgan, J. Power, and J. Waldron. Platform independent dynamic Java virtual machine analysis: the Java Grande Forum benchmark suite. In Proceedings of the 2001 joint ACM-ISCOPE Conference on Java Grande, 2001.

Digital Library

[15]

D. Detlefs and O. Agesen. Inlining of virtual methods. In Proceedings of the 13th European Conference on Object-oriented Programming (ECOOP), 1999.

Digital Library

[16]

S. Dieckmann and U. Holzle. A study of the allocation behavior of the SPECjvm98 java benchmarks. In Proceedings of the 13th European Conference on Object-oriented Programming (ECOOP), 1999.

Digital Library

[17]

I. Dragos and M. Odersky. Compiling generics through user-directed type specialization. In Proceedings of the 4th Workshop on the Implementation, Compilation, Optimization of Object-oriented Languages and Programming Systems (ICOOOLPS), 2009.

Digital Library

[18]

G. Dubochet and M. Odersky. Compiling structural types on the JVM: a comparison of reflective and generative techniques from Scalas perspective. In Proceedings of the 4th Workshop on the Implementation, Compilation, Optimization of Object-oriented Languages and Programming Systems (ICOOOLPS), 2009.

Digital Library

[19]

B. Dufour, K. Driesen, L. Hendren, and C. Verbrugge. Dynamic metrics for Java. In Proceedings of the 18th Conference on Object-oriented Programing, Systems, Languages, and Applications (OOPSLA), 2003.

Digital Library

[20]

Common Language Infrastructure (CLI). ECMA International, 5th edition, December 2010.

[21]

L. Eeckhout, A. Georges, and K. De Bosschere. How Java programs interact with virtual machines at the microarchitectural level. In Proceedings of the 18th Conference on Object-oriented Programing, Systems, Languages, and Applications (OOPSLA), 2003.

Digital Library

[22]

D. Gregg, J. Power, and J. Waldron. A method-level comparison of the Java Grande and SPEC JVM98 benchmark suites. Concurrency and Computation: Practice and Experience, 17: 757--773, 2005.

Digital Library

[23]

N. M. Hanish and W. E. Cohen. Hardware support for profiling Java programs. In Proceedings of the Workshop on Hardware Support for Objects and Microarchitectures for Java, 1999.

[24]

R. E. Hank, W.-M. W. Hwu, and B. R. Rau. Region-based compilation: an introduction and motivation. In Proceedings of the 28th International Symposium on Microarchitecture (MICRO), 1995.

Digital Library

[25]

U. Holzle, C. Chambers, and D. Ungar. Optimizing dynamically-typed object-oriented languages with polymorphic inline caches. In Proceedings of the 5th European Conference on Object-oriented Programming (ECOOP), 1991.

Digital Library

[26]

K. Hoste and L. Eeckhout. Microarchitecture-independent workload characterization. IEEE Micro, 27:63--72, 2007.

Digital Library

[27]

I. Jibaja, S. Blackburn, M. Haghighat, and K. McKinley. Deferred gratification: Engineering for high performance garbage collection from the get go. In Proceedings of the Workshop on Memory Systems Performance and Correctness (MSPC), 2011.

Digital Library

[28]

R. E. Jones and C. Ryder. A study of Java object demographics. In Proceedings of the 7th International Symposium on Memory Management (ISMM), 2008.

Digital Library

[29]

R. K. Karmani, A. Shali, and G. Agha. Actor frameworks for the JVM platform: a comparative analysis. In Proceedings of the 7th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2009.

Digital Library

[30]

G. Lashari and S. Srinivas. Characterizing Java application performance. In Proceedings of the 17th International Parallel and Distributed Processing Symposium (IPDPS), 2003.

Digital Library

[31]

T. Lindholm and F. Yellin. The Java Virtual Machine Specification. Addison-Wesley, 2nd edition, 1999.

Digital Library

[32]

A. McCallum, K. Schultz, and S. Singh. FACTORIE: Probabilistic programming via imperatively defined factor graphs. Advances on Neural Information Processing Systems, 2009.

[33]

P. Moret, W. Binder, and A. Villazon. CCCP: Complete calling context profiling in virtual execution environments. In Proceedings of the Workshop on Partial Evaluation and Program Manipulation (PEPM), 2009.

Digital Library

[34]

M. Odersky, L. Spoon, and B. Venners. Programming in Scala. Artima Press, 2nd edition, 2010.

[35]

K. Ogata, D. Mikurube, K. Kawachiya, S. Trent, and T. Onodera. A study of Javas non-Java memory. In Proceedings of the 25th Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2010.

Digital Library

[36]

K. Pearson. On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2:559--572, 1901.

[37]

D. Ramage, E. Rosen, J. Chuang, C. D. Manning, and D. A. McFarland. Topic modeling for the social sciences. In NIPS Workshop on Applications for Topic Models: Text and Beyond, 2009.

[38]

P. Ratanaworabhan, B. Livshits, and B. G. Zorn. JSMeter: comparing the behavior of JavaScript benchmarks with real Web applications. In Proceedings of the 2010 USENIX Conference on Web Application Development, 2010.

Digital Library

[39]

G. Richards, S. Lebresne, B. Burg, and J. Vitek. An analysis of the dynamic behavior of JavaScript programs. In Proceedings of the Conference on Programming Language Design and Implementation (PLDI), 2010.

Digital Library

[40]

I. Rogers, J. Zhao, and I. Watson. Approaches to reflective method invocation. In Proceedings of the 3rd Workshop on Implementation, Compilation, Optimization of Object-oriented Languages, Programs and Systems (ICOOOLPS), 2008.

[41]

J. R. Rose. Bytecodes meet combinators: invokedynamic on the JVM. In Proceedings of the 3rd Workshop on Virtual Machines and Intermediate Languages (VMIL), 2009.

Digital Library

[42]

A. Sarimbekov, P. Moret, W. Binder, A. Sewe, and M. Mezini. Complete and platform-independent calling context profiling for the Java virtual machine. In Proceedings of the 6th Workshop on Bytecode Semantics, Verification, Analysis and Transformation (BYTECODE), 2011.

Digital Library

[43]

A. Sarimbekov, A. Sewe, W. Binder, P. Moret, M. Schoberl, and M. Mezini. Portable and accurate collection of calling-context-sensitive bytecode metrics for the java virtual machine. In Proceedings of the 9th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2011.

Digital Library

[44]

M. Schinz. Compiling Scala for the Java Virtual Machine. PhD thesis, EPFL, Switzerland, September 2005.

[45]

A. Sewe. Scala ð Java mod JVM. In Proceedings of the Work-in-Progress Session at the 8th International Conference on the Principles and Practice of Programming in Java (PPPJ), volume 692 of CEUR Workshop Proceedings, 2010.

[46]

K. Shiv, K. Chow, Y. Wang, and D. Petrochenko. SPECjvm2008 performance characterization. In Proceedings of the SPEC Benchmark Workshop, 2009.

Digital Library

[47]

A. M. Sloane. Experiences with domain-specific language embedding in Scala. In Proceedings of the 2nd International Workshop on Domain-Specific Program Development (DSPD), 2008.

[48]

T. Suganuma, T. Yasue, and T. Nakatani. A region-based compilation technique for a Java just-in-time compiler. In Proceedings of the 16th Conference on Programming Language Design and Implementation (PLDI), 2003.

Digital Library

[49]

C. Thalinger and J. Rose. Optimizing invokedynamic. In Proceedings of the 8th International Conference on the Principles and Practice of Programming in Java (PPPJ), 2010.

Digital Library

Cited By

Vukasovic MProkopec A(2023)Exploiting Partially Context-Sensitive Profiles to Improve Performance of Hot CodeACM Transactions on Programming Languages and Systems10.1145/3612937Online publication date: 13-Sep-2023
https://doi.org/10.1145/3612937
Basso MRosà AOmini LBinder WVerbrugge CLhoták OShen X(2023)Java Vector API: Benchmarking and Performance AnalysisProceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction10.1145/3578360.3580265(1-12)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3578360.3580265
Aceto LAttard DFrancalanza AIngólfsdóttir A(2021)On Benchmarking for Concurrent Runtime VerificationFundamental Approaches to Software Engineering10.1007/978-3-030-71500-7_1(3-23)Online publication date: 20-Mar-2021
https://doi.org/10.1007/978-3-030-71500-7_1
Show More Cited By

Index Terms

Da capo con scala: design and analysis of a scala benchmark suite for the java virtual machine

Recommendations

A Comprehensive Java Benchmark Study on Memory and Garbage Collection Behavior of DaCapo, DaCapo Scala, and SPECjvm2008
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Benchmark suites are an indispensable part of scientific research to compare different approaches against each another. The diversity of benchmarks is an important asset to evaluate novel approaches for effectiveness and weaknesses. In this paper, we ...
Da capo con scala: design and analysis of a scala benchmark suite for the java virtual machine
OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications

Originally conceived as the target platform for Java alone, the Java Virtual Machine (JVM) has since been targeted by other languages, one of which is Scala. This trend, however, is not yet reflected by the benchmark suites commonly used in JVM ...
Performance and Energy-Efficiency of Scala on Mobile Devices
NGMAST '13: Proceedings of the 2013 Seventh International Conference on Next Generation Mobile Apps, Services and Technologies

The objective of this study was to compare how new languages like Scala perform on mobile environments compared to the classical languages. Since Scala also runs on the Java Virtual Machine (JVM), it is possible to run the code on Android devices and ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 46, Issue 10

OOPSLA '11

October 2011

1063 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/2076021

Issue’s Table of Contents

OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications
October 2011
1104 pages
ISBN:9781450309400
DOI:10.1145/2048066
General Chair:
Cristina Videira Lopes
University of California, Irvine, USA
,
Program Chair:
Kathleen Fisher
Tufts University, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2011

Published in SIGPLAN Volume 46, Issue 10

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

92
Total Citations
View Citations
691
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)5

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vukasovic MProkopec A(2023)Exploiting Partially Context-Sensitive Profiles to Improve Performance of Hot CodeACM Transactions on Programming Languages and Systems10.1145/3612937Online publication date: 13-Sep-2023
https://doi.org/10.1145/3612937
Basso MRosà AOmini LBinder WVerbrugge CLhoták OShen X(2023)Java Vector API: Benchmarking and Performance AnalysisProceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction10.1145/3578360.3580265(1-12)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3578360.3580265
Aceto LAttard DFrancalanza AIngólfsdóttir A(2021)On Benchmarking for Concurrent Runtime VerificationFundamental Approaches to Software Engineering10.1007/978-3-030-71500-7_1(3-23)Online publication date: 20-Mar-2021
https://doi.org/10.1007/978-3-030-71500-7_1
Rosà ABinder W(2020)$$\mathsf{P}^{3}$$: A Profiler Suite for Parallel Applications on the Java Virtual MachineProgramming Languages and Systems10.1007/978-3-030-64437-6_19(364-372)Online publication date: 24-Nov-2020
https://doi.org/10.1007/978-3-030-64437-6_19
Prokopec ADuboscq GLeopoldseder DWürthinger TKandemir MJimborean AMoseley T(2019)An optimization-driven incremental inline substitution algorithm for just-in-time compilersProceedings of the 2019 IEEE/ACM International Symposium on Code Generation and Optimization10.5555/3314872.3314893(164-179)Online publication date: 16-Feb-2019
https://dl.acm.org/doi/10.5555/3314872.3314893
Prokopec ARosà ALeopoldseder DDuboscq GTůma PStudener MBulej LZheng YVillazón ASimon DWürthinger TBinder WSmaragdakis Y(2019)Renaissance: a modern benchmark suite for parallel applications on the JVMProceedings Companion of the 2019 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity10.1145/3359061.3362778(11-12)Online publication date: 20-Oct-2019
https://dl.acm.org/doi/10.1145/3359061.3362778
Rosà ARosales EBinder W(2019)Analysis and Optimization of Task Granularity on the Java Virtual MachineACM Transactions on Programming Languages and Systems10.1145/333849741:3(1-47)Online publication date: 16-Jul-2019
https://dl.acm.org/doi/10.1145/3338497
Rosà ARosales ESchiavio FBinder WMarr SSartor J(2018)Understanding task granularity on the JVM: profiling, analysis, and optimizationCompanion Proceedings of the 2nd International Conference on the Art, Science, and Engineering of Programming10.1145/3191697.3191724(54-56)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.1145/3191697.3191724
Sarimbekov AStadler LBulej LSewe APodzimek AZheng YBinder W(2016)Workload characterization of JVM languagesSoftware—Practice & Experience10.1002/spe.233746:8(1053-1089)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1002/spe.2337
Hussain ICsallner CGrechanik MXie QPark STaneja KMainul Hossain B(2016)RUGRATSoftware—Practice & Experience10.1002/spe.229046:3(405-431)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1002/spe.2290
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents