Article

Automated, contract-based user testing of commercial-off-the-shelf components

Authors:

Lionel C. Briand,

Michal M. SówkaAuthors Info & Claims

ICSE '06: Proceedings of the 28th international conference on Software engineering

Pages 92 - 101

https://doi.org/10.1145/1134285.1134300

Published: 28 May 2006 Publication History

Abstract

Commercial-off-the-Shelf (COTS) components provide a means to construct software (component-based) systems in reduced time and cost. In a COTS component software market there exist component vendors (original developers of the component) and component users (developers of the component-based systems). The former provide the component to the user without source code or design documentation, and as a result it is difficult for the latter to adequately test the component when deployed in their system. In this article we propose a framework that clarifies the roles and responsibilities of both parties so that the user can adequately test the component in a deployment environment and the vendor does not need to release proprietary details. Then, based on this framework we combine and adapt two specification-based testing techniques and describe (and implement) a method for the automated generation of adequate test sets. An evaluation of our approach on a case study demonstrates that it is possible to automatically generate cost effective test sequences and that these test sequences are effective at detecting complex errors.

References

[1]

P. Ammann, A. J. Offutt and H. S. Hong, "Coverage Criteria for Logical Expressions," Proc. IEEE ISSRE, pp. 99--107, 2003.]]

Digital Library

[2]

J. H. Andrews, L. C. Briand and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?," Proc. IEEE ICSE, pp. 402--411, 2005.]]

Digital Library

[3]

S. Beydeda and V. Gruhn, "An Integrated Testing Technique for Component Based Software," Proc. IEEE ICCSA, 2001.]]

Digital Library

[4]

S. Beydeda and V. Gruhn, Testing COTS Components and Systems, Springer, 2005.]]

Digital Library

[5]

R. V. Binder, Testing Object-Oriented Systems - Models, Patterns, and Tools, Addison-Wesley, 1999.]]

Digital Library

[6]

R. G. Busacker and P. J. Gowen, "A Procedure for Determining a Family of Minimal-Cost Network Flow Patterns," John Hopkins University, O.R.O., Technical Report 15, 1961.]]

[7]

R. H. Carver and K.-C. Tai, "Use of Sequencing Constraints for Specification Based Testing of Concurrent Programs," IEEE TSE, vol. 24 (6), pp. 471--490, 1998.]]

Digital Library

[8]

A. Cechich and M. Polo, "COTS Component Testing through Aspect-Based Metadata," in S. Beydeda and V. Gruhn, Eds., Testing COTS Components and Systems, Springer, pp. 71--88, 2005.]]

[9]

N. Christofides, V. Campos, A. Corberan and E. Mota, "An algorithm for the rural postman problem on a directed graph," Mathematical Programming Studies, vol. 26, pp. 155--166, 1986.]]

[10]

F. J. Daniels and K.-C. Tai, "Measuring the Effectiveness of Method Test Sequences Derived from Sequencing Constraints," Proc.TOOLS, pp. 74--83, 1999.]]

Digital Library

[11]

J. L. Devore, Probability and Statistics for Eng. and the Sciences, Duxbury Press, 5th Edition, 1999.]]

[12]

M. Dror, Arc Routing: Theory, Solutions and Applications, Kluwer, 2000.]]

[13]

J. Edmonds, "Optimum branchings," Journal Research of the National Bureau of Standards 71B, pp. --33-240, 1967.]]

[14]

J. Z. Gao, H.-S. J. Tsao and Y. Wu, Testing and Quality Assurance for Component-based Software, Artech House, 2003.]]

Digital Library

[15]

S. Ghosh and A. P. Mathur, "Interface Mutation to assess the adequacy of tests for components and systems," Proc. TOOLS, pp.--37-46, 2000.]]

Digital Library

[16]

JBoss, JBoss Application Server, http://www.jboss.com (Last accessed 26 Jan. 2005).]]

[17]

B. Karçali and K.-C. Tai, "Automated test sequence generation using sequencing constraints for concurrent programs," Proceedings of the International Symposium on Software Engineering for Parallel and Distributed Systems, pp. 97--108, 1999.]]

Digital Library

[18]

S. Kim, J. A. Clark and J. A. McDermid, "Class Mutation: Mutation Testing for Object-Oriented Programs," Proc. Net.ObjectDays, 2000.]]

[19]

K. N. King and A. J. Offutt, "A Fortran Language System for Mutation-Based Soft. Testing," Software - Practice and Experience, vol. 21 (7), pp. --86-718, 1991.]]

Digital Library

[20]

V. Massol and T. Husted, JUnit in Action, Manning Publications, 2004.]]

Digital Library

[21]

OMG, "UML 2.0 Superstructure Specification," Object Management Group, Final Adopted Specification ptc/03-08-02, 2003.]]

[22]

A. Orso, M. J. Harrold, D. Rosenblum, G. Rothermel, M. L. Soffa and H. Do, "Using component metacontent to support the regression testing of component-based Software," Proc. IEEE ICSM, pp. 716--725, 2001.]]

Digital Library

[23]

T. J. Ostrand and M. J. Balcer, "The Category-Partition Method for Specifying and Generating Functional Test," Communications of the ACM, vol. 31 (6), pp. 676--686, 1988.]]

Digital Library

[24]

A. Polini and A. Bertolino, "A User-Oriented Framework for Component Deployment Testing," in S. Beydeda and V. Gruhn, Eds., Testing COTS Components and Systems, Springer, pp. 169--194, 2005.]]

[25]

D. S. Rosenblum, "Adequate testing of component-based software," Department of Information and Computer Science, University of California, Technical Report 97--34, 1997.]]

[26]

Sun Microsystems, Enterprise JavaBeans Technology, java.sun.com/products/ejb/ (Last accessed 26 Jan. 2005)]]

[27]

Sun Microsystems, Java Blueprints, java.sun.com/ reference/blueprints/ (Last accessed 26 Jan. 2005).]]

[28]

C. Szyperski, D. Gruntz and S. Murer, Component Software - Beyond Object-Oriented Programming, Addison-Wesley, 2nd Edition, 1999.]]

Digital Library

[29]

J. Warmer and A. Kleppe, The Object Constraint Language, Addison Wesley, 2nd Edition, 2003.]]

Digital Library

[30]

E. Weyuker, "Automatically Generating Test Data from a Boolean Specification," IEEE TSE, vol. 20 (5), pp. 53--363, 1994.]]

Digital Library

Cited By

Afzal ALe Goues CTimperley C(2022)Mithra: Anomaly Detection as an Oracle for Cyberphysical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2021.312068048:11(4535-4552)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TSE.2021.3120680
Chen JYin YCai SGeng YHuang L(2022)An Improved Test Case Generation Method based on Test Requirements for Testing Software Component2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C57518.2022.00039(209-218)Online publication date: Dec-2022
https://doi.org/10.1109/QRS-C57518.2022.00039
Namin AXue XRosas OSharma P(2015)MuRankerSoftware Testing, Verification & Reliability10.1002/stvr.154225:5-7(572-604)Online publication date: 1-Aug-2015
https://dl.acm.org/doi/10.1002/stvr.1542
Show More Cited By

Index Terms

Automated, contract-based user testing of commercial-off-the-shelf components
1. General and reference
  1. Cross-computing tools and techniques
    1. Reliability
    2. Validation
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Process validation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

A European COTS Architecture with Built-in Tests
OOIS '02: Proceedings of the 8th International Conference on Object-Oriented. Information Systems

This paper presents a European approach to the development of a new industrial architecture of commercial off-the-shelf (COTS) software components, and a practical technology for design and implementation of test-reusable COTS. This work, known as the ...
Hierarchical composition of industrial components
Special issue on new software composition concepts

In this paper we focus on the hierarchical composition of instances of arbitrary industrial component models yielding new (compound) components with specified capabilities and requirements which can themselves be composed to yield higher level ...
An integrated approach for developing semantic-mismatch free commercial off the shelf (COTS) components
FIT '10: Proceedings of the 8th International Conference on Frontiers of Information Technology

Software in the modern age are mostly developed by the integration of pre fabricated COTS components as it is the simplest way to develop systems quickly consuming lesser cost as compared to the traditional development approaches. The promising features ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '06: Proceedings of the 28th international conference on Software engineering

May 2006

1110 pages

ISBN:1595933751

DOI:10.1145/1134285

General Chair:
Leon J. Osterweil
University of Massachusetts, USA
,
Program Chairs:
Dieter Rombach
TU Kaiserslautern, Germany
,
Mary Lou Soffa
University of Virginia, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE06

Sponsor:

ICSE06: International Conference on Software Engineering

May 20 - 28, 2006

Shanghai, China

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
1,046
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Afzal ALe Goues CTimperley C(2022)Mithra: Anomaly Detection as an Oracle for Cyberphysical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2021.312068048:11(4535-4552)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TSE.2021.3120680
Chen JYin YCai SGeng YHuang L(2022)An Improved Test Case Generation Method based on Test Requirements for Testing Software Component2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C57518.2022.00039(209-218)Online publication date: Dec-2022
https://doi.org/10.1109/QRS-C57518.2022.00039
Namin AXue XRosas OSharma P(2015)MuRankerSoftware Testing, Verification & Reliability10.1002/stvr.154225:5-7(572-604)Online publication date: 1-Aug-2015
https://dl.acm.org/doi/10.1002/stvr.1542
Alves EMachado PRamalho F(2014)Automatic generation of built-in contract test driversSoftware and Systems Modeling (SoSyM)10.1007/s10270-012-0282-813:3(1141-1165)Online publication date: 1-Jul-2014
https://dl.acm.org/doi/10.1007/s10270-012-0282-8
Ye CJacobsen H(2013)Whitening SOA Testing via Event ExposureIEEE Transactions on Software Engineering10.1109/TSE.2013.2039:10(1444-1465)Online publication date: 1-Oct-2013
https://dl.acm.org/doi/10.1109/TSE.2013.20
Eler MMasiero P(2013)BISTFaSCProceedings of the 2013 VII Brazilian Symposium on Software Components, Architectures and Reuse10.1109/SBCARS.2013.20(89-98)Online publication date: 29-Sep-2013
https://dl.acm.org/doi/10.1109/SBCARS.2013.20
Namin AKakarla SDwyer MTip F(2011)The use of mutation in testing experiments and its sensitivity to external threatsProceedings of the 2011 International Symposium on Software Testing and Analysis10.1145/2001420.2001461(342-352)Online publication date: 17-Jul-2011
https://dl.acm.org/doi/10.1145/2001420.2001461
da Silva Cde Lemos RGiese HCheng B(2011)Dynamic plans for integration testing of self-adaptive software systemsProceedings of the 6th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1145/1988008.1988029(148-157)Online publication date: 23-May-2011
https://dl.acm.org/doi/10.1145/1988008.1988029
Eler MBertolino AMasiero P(2011)More testable service compositions by test metadataProceedings of the Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System Engineering10.1109/SOSE.2011.6139109(204-213)Online publication date: 12-Dec-2011
https://dl.acm.org/doi/10.1109/SOSE.2011.6139109
Zhang LHou SHu JXie TMei HKramer JBishop JDevanbu PUchitel S(2010)Is operator-based mutant selection superior to random mutant selection?Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 110.1145/1806799.1806863(435-444)Online publication date: 1-May-2010
https://dl.acm.org/doi/10.1145/1806799.1806863
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents