chapter

Service-Oriented Architectures Testing: A Survey

Authors:

Gerardo Canfora,

Massimiliano PentaAuthors Info & Claims

Software Engineering: International Summer Schools, ISSSE 2006-2008, Salerno, Italy, Revised Tutorial Lectures

January 2009

Pages 78 - 105

https://doi.org/10.1007/978-3-540-95888-8_4

Published: 22 January 2009 Publication History

Abstract

Testing of Service Oriented Architectures (SOA) plays a critical role in ensuring a successful deployment in any enterprise. SOA testing must span several levels, from individual services to inter-enterprise federations of systems, and must cover functional and non-functional aspects.

SOA unique combination of features, such as run-time discovery of services, ultra-late binding, QoS aware composition, and SLA automated negotiation, challenge many existing testing techniques. As an example, run-time discovery and ultra-late binding entail that the actual configuration of a system is known only during the execution, and this makes many existing integration testing techniques inadequate. Similarly, QoS aware composition and SLA automated negotiation means that a service may deliver with different performances in different contexts, thus making most existing performance testing techniques to fail.

Whilst SOA testing is a recent area of investigation, the literature presents a number of approaches and techniques that either extend traditional testing or develop novel ideas with the aim of addressing the specific problems of testing service-centric systems. This chapter reports a survey of recent research achievements related to SOA testing. Challenges are analyzed from the viewpoints of different stakeholders and solutions are presented for different levels of testing, including unit, integration, and regression testing. The chapter covers both functional and non-functional testing, and explores ways to improve the testability of SOA.

References

[1]

Paolucci, M., Kawamura, T., Payne, T.R., Sycara, K.: Semantic matching of web services capabilities. In: Horrocks, I., Hendler, J. (eds.) ISWC 2002. LNCS, vol. 2342, pp. 333-347. Springer, Heidelberg (2002)

Digital Library

[2]

Bromberg, Y.D., Issarny, V.: INDISS: Interoperable discovery system for networked services. In: Alonso, G. (ed.) Middleware 2005. LNCS, vol. 3790, pp. 164-183. Springer, Heidelberg (2005)

Digital Library

[3]

Pistore, M., Traverso, P.: Assumption-based composition and monitoring of web services. In: Test and Analysis of web Services, pp. 307-335. Springer, Heidelberg (2007)

[4]

Di Nitto, E., Di Penta, M., Gambi, A., Ripa, G., Villani, M.L.: Negotiation of service level agreements: An architecture and a search-based approach. In: Krämer, B.J., Lin, K.- J., Narasimhan, P. (eds.) ICSOC 2007. LNCS, vol. 4749, pp. 295-306. Springer, Heidelberg (2007)

[5]

Baresi, L., Guinea, S.: Towards dynamic monitoring ofWS-BPEL processes. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 269-282. Springer, Heidelberg (2005)

[6]

Kephart, J., Chess, D.: The vision of autonomic computing. IEEE Computer (2003)

[7]

Hinchey, M.G., Sterritt, R.: Self-managing software. Computer 39, 107-109 (2006)

Digital Library

[8]

Turner, M., Budgen, D., Brereton, P.: Turning software into a service. IEEE Computer 36, 38-44 (2003)

Digital Library

[9]

Walkerdine, J., Melville, L., Sommerville, I.: Dependability properties of P2P architectures. In: 2nd International Conference on Peer-to-Peer Computing (P2P 2002), Linköping, Sweden, 5-7 September 2002, pp. 173-174 (2002)

[10]

Baresi, L., Ghezzi, C., Guinea, S.: Smart Monitors for Composed Services. In: Proc. 2nd International Conference on Service Oriented Computing (ICSOC 2004), pp. 193-202. ACM, New York (2004)

[11]

Canfora, G., Di Penta, M.: Testing services and service-centric systems: Challenges and opportunities. IT Professional 8, 10-17 (2006)

Digital Library

[12]

Milanova, A., Rountev, A., Ryder, B.G.: Parameterized object sensitivity for points-to analysis for Java. ACM Trans. Softw. Eng. Methodol. 14, 1-41 (2005)

Digital Library

[13]

Bruno, M., Canfora, G., Di Penta, M., Esposito, G., Mazza, V.: Using test cases as contract to ensure service compliance across releases. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 87-100. Springer, Heidelberg (2005)

[14]

Di Penta, M., Bruno, M., Esposito, G., Mazza, V., Canfora, G.: Web services regression testing. In: Baresi, L., Nitto, E.D. (eds.) Test and Analysis of web Services, pp. 205-234. Springer, Heidelberg (2007)

[15]

Ernst, M.D., Cockrell, J., Griswold, W.G., Notkin, D.: Dynamically discovering likely program invariants to support program evolution. IEEE Trans. Software Eng. 27, 99-123 (2001)

Digital Library

[16]

Lorenzoli, D., Mariani, L., Pezzè, M.: Automatic generation of software behavioral models. In: 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008, pp. 501-510 (2008)

Digital Library

[17]

Ghezzi, C., Mocci, A., Monga, M.: Efficient recovery of algebraic specifications for stateful components. In: IWPSE 2007: Ninth international workshop on Principles of software evolution, pp. 98-105. ACM, New York (2007)

Digital Library

[18]

McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14, 105-156 (2004)

Digital Library

[19]

Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Information & Software Technology 43, 841-854 (2001)

[20]

Di Penta, M., Canfora, G., Esposito, G., Mazza, V., Bruno, M.: Search-based testing of service level agreements. In: Proceedings of Genetic and Evolutionary Computation Conference, GECCO 2007, London, England, UK, July 7-11, 2007, pp. 1090-1097 (2007)

[21]

Bertolino, A., Gao, J., Marchetti, E., Polini, A.: Systematic generation of XML instances to test complex software applications. In: Guelfi, N., Buchs, D. (eds.) RISE 2006. LNCS, vol. 4401, pp. 114-129. Springer, Heidelberg (2007)

[22]

Bertolino, A., Gao, J., Marchetti, E., Polini, A.: TAXI - a tool for XML-based testing. In: 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007, pp. 53-54 (2007)

[23]

Ostrand, T., Balcer, M.: The category-partition method for specifying and generating functional tests. Communications of the Association for Computing Machinery 31 (1988)

Digital Library

[24]

Bai, X., Dong, W., Tsai, W.T., Chen, Y.: Wsdl-based automatic test case generation for web services testing. In: IEEE International Workshop on Service-Oriented System Engineering (SOSE), pp. 215-220. IEEE Computer Society, Los Alamitos (2005)

[25]

Conroy, K., Grechanik, M., Hellige, M., Liongosari, E., Xie, Q.: Automatic test generation from GUI applications for testing Web services. In: IEEE International Conference on Software Maintenance, ICSM 2007, pp. 345-354 (2007)

[26]

Li, Z., Sun, W., Jiang, Z.B., Zhang, X.: BPEL4WS unit testing: Framework and implementation. In: 2005 IEEE International Conference on web Services (ICWS 2005), Orlando, FL, USA, 11-15 July 2005, pp. 103-110 (2005)

[27]

Yuan, Y., Li, Z., Sun, W.: A graph-search based approach to BPEL4WS test generation. In: Proceedings of the International Conference on Software Engineering Advances (ICSEA 2006), Papeete, Tahiti, French Polynesia, October 28 - November 2, 2006, p. 14 (2006)

[28]

Tsai, W.T., Chen, Y., Paul, R.A., Liao, N., Huang, H.: Cooperative and group testing in verification of dynamic composite web services. In: 28th International Computer Software and Applications Conference (COMPSAC 2004), Design and Assessment of Trustworthy Software-Based Systems, Hong Kong, China, Proceedings, 27-30 September 2004, pp. 170- 173 (2004)

[29]

Bucchiarone, A., Melgratti, H., Severoni, F.: Testing service composition. In: Proceedings of the 8th Argentine Symposium on Software Engineering (ASSE 2007) (2007)

[30]

Canfora, G., Di Penta, M., Esposito, R., Villani, M.L.: A framework for QoS-aware binding and re-binding of composite Web services. Journal of Systems and Software (in press, 2008)

[31]

Canfora, G., Di Penta, M., Esposito, R., Villani, M.L.: An approach for QoS-aware service composition based on genetic algorithms. In: Genetic and Evolutionary Computation Conference, GECCO 2005, Proceedings, Washington DC, USA, June 25-29, 2005, pp. 1069-1075. ACM Press, New York (2005)

Digital Library

[32]

Binder, R.V.: Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley Publishing Company, Reading (2000)

[33]

Tsai, W.T., Paul, R.J., Song, W., Cao, Z.: Coyote: An XML-based framework for Web services testing. In: 7th IEEE International Symposium on High-Assurance Systems Engineering (HASE 2002), Tokyo, Japan, 23-25 October 2002, pp. 173-176 (2002)

[34]

Bertolino, A., Polini, A.: The audition framework for testing Web services interoperability. In: 31st EUROMICRO Conference on Software Engineering and Advanced Applications (EUROMICRO-SEAA 2005), Porto, Portugal, 30 August - 3 September 2005, pp. 134-142. IEEE Computer Society, Los Alamitos (2005)

[35]

Mei, L., Chan, W.K., Tse, T.H.: Data flow testing of service-oriented workflow applications. In: 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008, pp. 371-380 (2008)

Digital Library

[36]

Bartolini, C., Bertolino, A., Marchetti, E., Parissis, I.: Data flow-based validation of web services compositions: Perspectives and examples. In: de Lemos, V.R., Di Giandomenico, F., Muccini, H., Gacek, C., Vieira, M. (eds.) Architecting Dependable Systems. Springer, Heidelberg (2008)

[37]

Walkerdine, J., Hutchinson, J., Sawyer, P., Dobson, G., Onditi, V.: A faceted approach to service specification. In: International Conference on Internet and web Applications and Services (ICIW 2007), Le Morne, Mauritius, May 13-19, 2007, p. 20 (2007)

Digital Library

[38]

Colombo, M., Di Nitto, E., Di Penta, M., Distante, D., Zuccalà, M.: Speaking a common language: A conceptual model for describing service-oriented systems. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 48-60. Springer, Heidelberg (2005)

[39]

Ricca, F., Torchiano, M., Di Penta, M., Mariano Ceccato, P.T.: Using acceptance tests as a support for clarifying requirements: A series of experiments. Information and Software Technology (in press, 2008)

[40]

Mugridge, R., Cunningham, W.: Fit for Developing Software: Framework for Integrated Tests. Prentice-Hall, Englewood Cliffs (2005)

[41]

Dai, G., Bai, X., Wang, Y., Dai, F.: Contract-based testing for web services. In: 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), Beijing, China, 24-27 July 2007, pp. 517-526 (2007)

Digital Library

[42]

Meyer, B.: Object-Oriented Software Construction, 2nd edn. Prentice-Hall, Englewood Cliffs (1997)

[43]

Ruth, M., Tu, S.: Towards automating regression test selection forWeb services. In: Proceedings of the 16th International Conference on WorldWide Web, WWW2007, Banff, Alberta, Canada, May 8-12, 2007, pp. 1265-1266 (2007)

[44]

Ruth, M., Oh, S., Loup, A., Horton, B., Gallet, O., Mata, M., Tu, S.: Towards automatic regression test selection for Web services. In: 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), Beijing, China, 24-27 July 2007, pp. 729- 736 (2007)

Digital Library

[45]

Rothermel, G., Harrold, M.J.: A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Methodol. 6, 173-210 (1997)

Digital Library

[46]

Martin, E., Basu, S., Xie, T.: WebSob: A tool for robustness testing of web services. In: 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007, pp. 65-66 (2007)

[47]

Martin, E., Basu, S., Xie, T.: Automated testing and response analysis ofweb services. In: 2007 IEEE International Conference on web Services (ICWS 2007), Salt Lake City, Utah, USA, July 9-13, 2007, pp. 647-654 (2007)

[48]

Csallner, C., Smaragdakis, Y.: JCrasher: an automatic robustness tester for Java. Softw. Pract. Exper. 34, 1025-1050 (2004)

Digital Library

[49]

Fu, C., Ryder, B.G., Milanova, A., Wonnacott, D.: Testing of Java Web services for robustness. In: Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis, ISSTA2004, Boston, Massachusetts, USA, July 11-14, 2004, pp. 23-34 (2004)

Digital Library

[50]

Canfora, G., Di Penta, M., Esposito, R., Perfetto, F., Villani, M.L.: Service composition (re)Binding driven by application-specific qoS. In: Dan, A., Lamersdorf, W. (eds.) ICSOC 2006. LNCS, vol. 4294, pp. 141-152. Springer, Heidelberg (2006)

[51]

Voas, J.M.: Fault injection for the masses. IEEE Computer 30, 129-130 (1997)

Digital Library

[52]

Offutt, J., Xu, W.: Generating test cases for Web services using data perturbation. In: SIGSOFT Softw. Eng. Notes - SECTION:Workshop on testing, analysis and verification ofWeb services (TAV-WEB), vol. 29, pp. 1-10 (2004)

[53]

Beizer, B.: Software Testing Techniques, 2nd edn. International Thomson Computer Press (1990)

[54]

Looker, N., Munro, M., Xu, J.: Ws-fit: A tool for dependability analysis of Web services. In: Proceedings of 28th International Computer Software and Applications Conference (COMPSAC 2004), Design and Assessment of Trustworthy Software-Based Systems, Hong Kong, China, 27-30 September 2004, pp. 120-123 (2004)

[55]

Tsai, W.T., Gao, J., Wei, X., Chen, Y.: Testability of software in service-oriented architecture. In: 30th Annual International Computer Software and Applications Conference (COMPSAC 2006), Chicago, Illinois, USA, 17-21 September 2006, pp. 163-170 (2006)

Digital Library

[56]

Tsai, W.T., Paul, R.J., Wang, Y., Fan, C., Wang, D.: Extending WSDL to facilitate Web services testing. In: 7th IEEE International Symposium on High-Assurance Systems Engineering (HASE 2002), Tokyo, Japan, 23-25 October 2002, pp. 171-172 (2002)

[57]

Tsai, W.T., Paul, R.J., Cao, Z., Yu, L., Saimi, A.: Verification of Web services using an enhanced UDDI server. In: Proceedings of the Eighth International Workshop on Object-Oriented Real-Time Dependable Systems, pp. 131-138 (2003)

[58]

Heckel, R., Mariani, L.: Automatic conformance testing of web services. In: Cerioli, M. (ed.) FASE 2005. LNCS, vol. 3442, pp. 34-48. Springer, Heidelberg (2005)

[59]

Heckel, R., Lohmann, M.: Towards contract-based testing of web services. Electr. Notes Theor. Comput. Sci. 116, 145-156 (2005)

Digital Library

[60]

Bai, X., Xu, D., Dai, G.: Dynamic reconfigurable testing of service-oriented architecture. In: 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), Beijing, China, 24-27 July 2007, pp. 368-378 (2007)

Digital Library

[61]

Bertolino, A., De Angelis, G., Polini, A.: A QoS test-bed generator for web services. In: Baresi, L., Fraternali, P., Houben, G.-J. (eds.) ICWE 2007. LNCS, vol. 4607, pp. 17-31. Springer, Heidelberg (2007)

Digital Library

[62]

Bertolino, A., De Angelis, G., Frantzen, L., Polini, A.: Model-Based Generation of Testbeds for Web Services. In: Suzuki, K., Higashino, T., Ulrich, A., Hasegawa, T. (eds.) Test-Com/FATES 2008. LNCS, vol. 5047, pp. 266-282. Springer, Heidelberg (2008)

[63]

Canfora, G., Di Penta, M.: SOA: Testing and self-checking. In: Keynote speech at the International Workshop on Web Services - Modeling and Testing (WS-MATE 2006) (2006)

[64]

Forestiero, A., Mastroianni, C., Papadakis, H., Fragopoulou, P., Troisi, A., Zimeo, E.: A scalable architecture for discovery and composition in P2P service networks. In: Grid Computing: Achievements and Prospects. Springer, Heidelberg (2008)

Cited By

Golmohammadi AZhang MArcuri A(2023)Testing RESTful APIs: A SurveyACM Transactions on Software Engineering and Methodology10.1145/361717533:1(1-41)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3617175
Marculescu BZhang MArcuri A(2022)On the Faults Found in REST APIs by Automated Test GenerationACM Transactions on Software Engineering and Methodology10.1145/349103831:3(1-43)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3491038
Chen LYu XWu JYang H(2021)CIPCMobile Information Systems10.1155/2021/29122402021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/2912240
Show More Cited By

Recommendations

Bringing white-box testing to Service Oriented Architectures through a Service Oriented Approach

The attractive feature of Service Oriented Architecture (SOA) is that pieces of software conceived and developed by independent organizations can be dynamically composed to provide richer functionality. The same reasons that enable flexible compositions,...
Automatic service composition with heterogeneous service-oriented architectures
ICOST'10: Proceedings of the Aging friendly technology for health and independence, and 8th international conference on Smart homes and health telematics

Service-Oriented Architecture is widely used to program pervasive spaces such as Smart Homes because of its capabilities to handle dynamic and heterogeneous environments. It is often the case that the services required are designed and implemented using ...
Introducing service-oriented coverage testing
ASE'08: Proceedings of the 23rd IEEE/ACM International Conference on Automated Software Engineering

Testing of service-oriented systems is challenged by loose coupling and high dynamism, which are the founding characteristics of this emerging paradigm. Traditional test approaches needs to be revised for such systems, and in particular white-box ...

Comments

Information & Contributors

Information

Published In

cover image Guide books

Software Engineering: International Summer Schools, ISSSE 2006-2008, Salerno, Italy, Revised Tutorial Lectures

January 2009

256 pages

ISBN:9783540958871

Editors:
Andrea Lucia
Dipartimento di Matematica e Informatica, Università di Salerno, Fisciano, Italy 84084
,
Filomena Ferrucci
Dipartimento di Matematica e Informatica (DMI), Università di Salerno, Fisciano, Italy 84084

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 22 January 2009

Qualifiers

Chapter

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Golmohammadi AZhang MArcuri A(2023)Testing RESTful APIs: A SurveyACM Transactions on Software Engineering and Methodology10.1145/361717533:1(1-41)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3617175
Marculescu BZhang MArcuri A(2022)On the Faults Found in REST APIs by Automated Test GenerationACM Transactions on Software Engineering and Methodology10.1145/349103831:3(1-43)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3491038
Chen LYu XWu JYang H(2021)CIPCMobile Information Systems10.1155/2021/29122402021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/2912240
Sahin OAkay B(2021)A Discrete Dynamic Artificial Bee Colony with Hyper-Scout for RESTful web service API test suite generationApplied Soft Computing10.1016/j.asoc.2021.107246104:COnline publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1016/j.asoc.2021.107246
Driessen S(2020)Staking Assets Management on Blockchains: Vision and RoadmapService-Oriented Computing – ICSOC 2020 Workshops10.1007/978-3-030-76352-7_1(3-9)Online publication date: 14-Dec-2020
https://dl.acm.org/doi/10.1007/978-3-030-76352-7_1
Zhang MMarculescu BArcuri AAuger AStützle T(2019)Resource-based test case generation for RESTful web servicesProceedings of the Genetic and Evolutionary Computation Conference10.1145/3321707.3321815(1426-1434)Online publication date: 13-Jul-2019
https://dl.acm.org/doi/10.1145/3321707.3321815
Arcuri A(2019)RESTful API Automated Test Case Generation with EvoMasterACM Transactions on Software Engineering and Methodology10.1145/329345528:1(1-37)Online publication date: 9-Jan-2019
https://dl.acm.org/doi/10.1145/3293455
Martínez-Fernández SFranch XBisbal J(2017)MercuryScience of Computer Programming10.1016/j.scico.2015.11.009134:C(61-74)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1016/j.scico.2015.11.009
Herbold SHoffmann A(2017)Model-based testing as a serviceInternational Journal on Software Tools for Technology Transfer (STTT)10.1007/s10009-017-0449-219:3(271-279)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1007/s10009-017-0449-2
Besson FMoura PKon FMilojicic D(2015)Bringing Test-Driven Development to web service choreographiesJournal of Systems and Software10.1016/j.jss.2014.09.03499:C(135-154)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.1016/j.jss.2014.09.034
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten