research-article

Automated performance assessment for service-oriented middleware: a case study on BPEL engines

Authors:

Domenico Bianculli,

Mauro Luigi DragoAuthors Info & Claims

WWW '10: Proceedings of the 19th international conference on World wide web

Pages 141 - 150

https://doi.org/10.1145/1772690.1772706

Published: 26 April 2010 Publication History

Abstract

Middleware for Web service compositions, such as BPEL engines, provides the execution environment for services as well as additional functionalities, such as monitoring and self-tuning. Given its role in service provisioning, it is very important to assess the performance of middleware in the context of a SOA. This paper presents SOABench, a framework for the automatic generation and execution of testbeds for benchmarking middleware for composite Web services and for assessing the performance of existing SOA infrastructures. SOABench defines a testbed model characterized by the composite services to execute, the workload to generate, the deployment configuration to use, the performance metrics to gather, the data analyses to perform on them, and the reports to produce. We have validated SOABench by benchmarking the performance of different BPEL engines.

References

[1]

L. Baresi, E. D. Nitto, and C. Ghezzi. Toward open-world software: Issue and challenges. IEEE Computer, 39(10):36--43, 2006.

Digital Library

[2]

C. Bartolini, A. Bertolino, E. Marchetti, and A. Polini. WS-TAXI: A WSDL-based testing tool for web services. In Proc. of ICST 2009, pages 326--335. IEEE Computer Society, 2009.

Digital Library

[3]

A. Bertolino, G. De Angelis, L. Frantzen, and A. Polini. Model-based generation of testbeds for web services. In Proc. of TESTCOM/FATES 2008, volume 5047 of LNCS, pages 266--282. Springer, 2008.

Digital Library

[4]

D. Bianculli, W. Binder, L. Drago, and C. Ghezzi. Transparent reputation management for composite web services. In Proc. of ICWS 2008, pages 621--628. IEEE Computer Society, 2008.

Digital Library

[5]

J. Bosch, S. Friedrichs, S. Jung, J. Helbig, and A. Scherdin. Service orientation in the enterprise. IEEE Computer, 40(11):51--56, 2007.

Digital Library

[6]

P. Budny, S. Govindharaj, and K. Schwan. Worldtravel: A testbed for service-oriented applications. In Proc. of ICSOC 2008, volume 5364 of LNCS, pages 438--452. Springer, 2008.

Digital Library

[7]

T. Erl. Service-Oriented Architecture: Concepts, Technology, and Design. Prentice Hall, 2005.

Digital Library

[8]

A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous Java performance evaluation. In Proc. of OOPSLA 2007, pages 57--76. ACM, 2007.

Digital Library

[9]

J. L. Hennessy and D. A. Patterson. Computer Architecture; A Quantitative Approach. Morgan Kaufmann Publishers Inc., 1992.

Digital Library

[10]

L. Juszczyk, H.-L. Truong, and S. Dustdar. GENESIS - a framework for automatic generation and steering of testbeds of complex web services. In Proc. of ICECCS 2008, pages 131--140. IEEE Computer Society, 2008.

Digital Library

[11]

E. D. Lazowska, J. Zahorjan, G. S. Graham, and K. C. Sevcik. Quantitative system performance: computer system analysis using queueing network models. Prentice-Hall, 1984.

Digital Library

[12]

L. Leonini, E. Riviere, and P. Felber. SPLAY: distributed systems evaluation made simple (or how to turn ideas into live systems in a breeze). In Proc. of NSDI'09, pages 185--198. USENIX Association, 2009.

Digital Library

[13]

E. Martin, S. Basu, and T. Xie. Automated testing and response analysis of web services. In Proc. of ICWS 2007, pages 647--654, 2007.

[14]

A. Mosincat and W. Binder. Transparent runtime adaptability for BPEL processes. In Proc. of ICSOC 2008, volume 5364 of LNCS, pages 241--255. Springer, 2008.

Digital Library

[15]

OASIS. UDDI specifications, 2004.

[16]

OASIS. Web Service Business Process Execution Language Version 2.0 Specification, 2007.

[17]

M. P. Papazoglou, P. Traverso, S. Dustdar, and F. Leymann. Service-oriented computing: State of the art and research challenges. IEEE Computer, 40(11):38--45, 2007.

Digital Library

[18]

W. T. Tsai, J. Gao, X. Wei, and Y. Chen. Testability of software in service-oriented architecture. In COMPSAC 2006, pages 163--170. IEEE Computer Society, 2006.

Digital Library

[19]

Y. Wang, M. J. Rutherford, A. Carzaniga, and A. L. Wolf. Automating experimentation on distributed testbeds. In Proc. of ASE 2005, pages 164--173. ACM, 2005.

Digital Library

Cited By

Lenhard JFerme VHarrer SGeiger MPautasso C(2018)Lessons Learned from Evaluating Workflow Management SystemsService-Oriented Computing – ICSOC 2017 Workshops10.1007/978-3-319-91764-1_17(215-227)Online publication date: 16-Jun-2018
https://doi.org/10.1007/978-3-319-91764-1_17
Harrer SLenhard JKopp OFerme VPautasso C(2017)A Pattern Language for Workflow Engine Conformance and Performance BenchmarkingProceedings of the 22nd European Conference on Pattern Languages of Programs10.1145/3147704.3147705(1-46)Online publication date: 12-Jul-2017
https://dl.acm.org/doi/10.1145/3147704.3147705
Ferme VSkouradaki MIvanchikj APautasso CLeymann F(2017)Performance Comparison Between BPMN 2.0 Workflow Management Systems VersionsEnterprise, Business-Process and Information Systems Modeling10.1007/978-3-319-59466-8_7(103-118)Online publication date: 17-May-2017
https://doi.org/10.1007/978-3-319-59466-8_7
Show More Cited By

Index Terms

Automated performance assessment for service-oriented middleware: a case study on BPEL engines

Recommendations

SOABench: performance evaluation of service-oriented middleware made easy
ICSE '10: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2

SOABench is a framework for the automatic generation, execution and analysis of testbeds for evaluating the performance of service-oriented middleware. Testbeds can be characterized in terms of the composite services to execute, the workload to generate,...
Service Oriented Middleware: Trends and Challenges
ITNG '10: Proceedings of the 2010 Seventh International Conference on Information Technology: New Generations

Service Oriented Architecture (SOA) is not only an architecture, rather it is a relationship between the service provider, broker and user. Designing middleware platforms for this architecture really helps in achieving interoperability, loose coupling ...
Middleware services for web service compositions
WWW '05: Special interest tracks and posters of the 14th international conference on World Wide Web

WS-* specifications cover a variety of issues ranging from security and reliability to transaction support in web services. However, these specifications do not address web service compositions. On the other hand, BPEL as the future standard web service ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

WWW '10: Proceedings of the 19th international conference on World wide web

April 2010

1407 pages

ISBN:9781605587998

DOI:10.1145/1772690

General Chairs:
Michael Rappa
North Carolina State University, USA
,
Paul Jones
University of North Carolina at Chapel Hill, USA
,
Program Chairs:
Juliana Freire
University of Utah, USA
,
Soumen Chakrabarti
Indian Institute of Technology, India

Copyright © 2010 International World Wide Web Conference Committee (IW3C2).

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WWW '10

WWW '10: The 19th International World Wide Web Conference

April 26 - 30, 2010

North Carolina, Raleigh, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
590
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lenhard JFerme VHarrer SGeiger MPautasso C(2018)Lessons Learned from Evaluating Workflow Management SystemsService-Oriented Computing – ICSOC 2017 Workshops10.1007/978-3-319-91764-1_17(215-227)Online publication date: 16-Jun-2018
https://doi.org/10.1007/978-3-319-91764-1_17
Harrer SLenhard JKopp OFerme VPautasso C(2017)A Pattern Language for Workflow Engine Conformance and Performance BenchmarkingProceedings of the 22nd European Conference on Pattern Languages of Programs10.1145/3147704.3147705(1-46)Online publication date: 12-Jul-2017
https://dl.acm.org/doi/10.1145/3147704.3147705
Ferme VSkouradaki MIvanchikj APautasso CLeymann F(2017)Performance Comparison Between BPMN 2.0 Workflow Management Systems VersionsEnterprise, Business-Process and Information Systems Modeling10.1007/978-3-319-59466-8_7(103-118)Online publication date: 17-May-2017
https://doi.org/10.1007/978-3-319-59466-8_7
Lenhard JWirtz G(2016)Portability of executable service-oriented processesService Oriented Computing and Applications10.1007/s11761-016-0195-410:4(391-411)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1007/s11761-016-0195-4
Skouradaki MFerme VPautasso CLeymann Fvan Hoorn A(2016)Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow PatternsAdvanced Information Systems Engineering10.1007/978-3-319-39696-5_5(67-82)Online publication date: 21-May-2016
https://doi.org/10.1007/978-3-319-39696-5_5
Geiger MHarrer SLenhard JCasar MVorndran AWirtz G(2015)BPMN Conformance in Open Source EnginesProceedings of the 2015 IEEE Symposium on Service-Oriented System Engineering10.1109/SOSE.2015.22(21-30)Online publication date: 30-Mar-2015
https://dl.acm.org/doi/10.1109/SOSE.2015.22
Ferme VIvanchikj APautasso C(2015)A Framework for Benchmarking BPMN 2.0 Workflow Management SystemsProceedings of the 13th International Conference on Business Process Management - Volume 925310.1007/978-3-319-23063-4_18(251-259)Online publication date: 31-Aug-2015
https://dl.acm.org/doi/10.1007/978-3-319-23063-4_18
Harrer SRöck CWirtz G(2014)Automated and Isolated Tests for Complex Middleware ProductsProceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops10.1109/ICSTW.2014.45(390-398)Online publication date: 31-Mar-2014
https://dl.acm.org/doi/10.1109/ICSTW.2014.45
Wagner SRoller DKopp OUnger TLeymann F(2013)Performance Optimizations for Interacting Business Processes*Proceedings of the 2013 IEEE International Conference on Cloud Engineering10.1109/IC2E.2013.34(210-216)Online publication date: 25-Mar-2013
https://dl.acm.org/doi/10.1109/IC2E.2013.34
Lenhard JWirtz G(2013)Measuring the Portability of Executable Service-Oriented ProcessesProceedings of the 2013 17th IEEE International Enterprise Distributed Object Computing Conference10.1109/EDOC.2013.21(117-126)Online publication date: 9-Sep-2013
https://dl.acm.org/doi/10.1109/EDOC.2013.21
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

EPUB

View this article in ePub.

Figures

Tables

Media

View Table of Conten