research-article

MD-ART: a test case generation method without test oracle problem

Authors:

Song HuangAuthors Info & Claims

SCTDCP 2016: Proceedings of the 1st International Workshop on Specification, Comprehension, Testing, and Debugging of Concurrent Programs

Pages 27 - 34

https://doi.org/10.1145/2975954.2975959

Published: 03 September 2016 Publication History

Abstract

Adaptive random testing (ART), as an improved random testing method, preserves the advantages of traditional random test method and overcomes the blindness of traditional random testing method. But it is usually not easy to validate the correctness of the output, except for some special test cases. In other words, the test oracle problem is unresolved. In this research, we introduced metamorphic testing (MT) and metamorphic distance into ART, which is called metamorphic distance based ART (MD-ART) to provide the test oracle. The results of primary experiment results show that MD-ART performs better than traditional MT and ART not only in test effectiveness but also in test efficiency and test coverage.

References

[1]

E.T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo, The Oracle Problem in Software Testing: A Survey, IEEE Transactions on Software Engineering, 41 (2015) 507-525.

Digital Library

[2]

E.J. Weyuker, The Oracle Assumption of Program Testing, in: Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, 1980, pp. 44-49

[3]

T.Y. Chen, S.C. Cheung, S. Yiu, Metamorphic testing: a new approach for generating next test cases, Department of Computer Science, Hong Kong University of Science and Technology, Tech. Rep. HKUST-CS98-01, (1998).

[4]

J. Zhang, J.-j. Chen, D. Hao, Y.-f. Xiong, B. Xie, L. Zhang, H. Mei, Search-Based Inference of Polynomial Metamorphic Relations, in: Proceedings of the 29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014), 2014, pp. 701-712.

Digital Library

[5]

U. Kanewala, Techniques for Automatic Detection of Metamorphic Relations, in: Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, 2014, pp. 237-238.

Digital Library

[6]

G. Singh, An Automated Metamorphic Testing Technique for Designing Effective Metamorphic Relations, in: Contemporary Computing, Springer, 2012, pp. 152-163.

[7]

Y. Cao, Z.Q. Zhou, T.Y. Chen, On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions, in: Quality Software (QSIC), 2013 13th International Conference on, IEEE, 2013, pp. 153-162.

Digital Library

[8]

L. Huai, K. Fei-Ching, D. Towey, C. Tsong Yueh, How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?, Software Engineering, IEEE Transactions on, 40 (2014) 4-22.

Digital Library

[9]

J. Mayer, R. Guderlei, An empirical study on the selection of good metamorphic relations, in: Computer Software and Applications Conference, 2006. COMPSAC'06. 30th Annual International, IEEE, 2006, pp. 475-484.

Digital Library

[10]

Z.W. Hui, S. Huang, A Formal Model for Metamorphic Relation Decomposition, in: 2013 Fourth World Congress on Software Engineering, Hong Kong, China, 2013, pp. 64- 68.

Digital Library

[11]

W.E. Wong, V. Debroy, A survey of software fault localization, Department of Computer Science, University of Texas at Dallas, Tech. Rep. UTDCS-45-09, (2009).

[12]

T.Y. Chen, F.-C. Kuo, Y. Liu, A. Tang, Metamorphic Testing and Testing with Special Values, in: SNPD, 2004, pp. 128- 134.

[13]

P. Wu, X.-C. Shi, J.-J. Tang, H.-M. Lin, T.Y. Chen, Metamorphic Testing and Special Case Testing: A Case Study, Journal of Software, 16 (2005) 1210-1220.

[14]

M. Jiang, T.Y. Chen, F.-C. Kuo, Z. Ding, Testing Central Processing Unit scheduling algorithms using Metamorphic Testing, in: Software Engineering and Service Science (ICSESS), 2013 4th IEEE International Conference on, IEEE, 2013, pp. 530-536.

[15]

A. Singh, S. Kang, Metamorphic Testing: Using the Properties of Sut, International Journal of Computer Technology and Applications 02 (2011) 1334-1336.

[16]

S. Yoo, Metamorphic testing of stochastic optimisation, in: Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, IEEE, 2010, pp. 192-201.

Digital Library

[17]

F.-C. Kuo, S. Liu, T.Y. Chen, Testing a binary space partitioning algorithm with metamorphic testing, in: Proceedings of the 2011 ACM Symposium on Applied Computing, ACM, 2011, pp. 1482-1489.

Digital Library

[18]

T.Y. Chen, F.-C. Kuo, R.G. Merkel, T.H. Tse, Adaptive Random Testing: The ART of test case diversity, J. Syst. Softw., 83 (2010) 60-66.

Digital Library

[19]

T.Y. Chen, H. Leung, I. Mak, Adaptive random testing, in: Advances in Computer Science-ASIAN 2004. Higher-Level Decision Making, Springer, 2005, pp. 320-329.

Digital Library

[20]

K.P. Chan, T.Y. Chen, F.-C. Kuo, D. Towey, A revisit of adaptive random testing by restriction, in: Computer Software and Applications Conference, 2004. COMPSAC 2004. Proceedings of the 28th Annual International, IEEE, 2004, pp. 78-85.

Digital Library

[21]

T.Y. Chen, R. Merkel, P. Wong, G. Eddy, Adaptive random testing through dynamic partitioning, in: Quality Software, 2004. QSIC 2004. Proceedings. Fourth International Conference on, IEEE, 2004, pp. 79-86.

Digital Library

[22]

T.Y. Chen, F.-C. Kuo, R.G. Merkel, T. Tse, Adaptive random testing: The art of test case diversity, Journal of Systems and Software, 83 (2010) 60-66.

Digital Library

[23]

T.Y. Chen, R. Merkel, An upper bound on software testing effectiveness, ACM Transactions on Software Engineering and Methodology (TOSEM), 17 (2008) 16.

Digital Library

[24]

R. Merkel, Analysis and enhancements of adaptive random testing, Ph. D. Thesis, (2005).

[25]

P. Wu, Iterative metamorphic testing, in: Computer Software and Applications Conference, 2005. COMPSAC 2005. 29th Annual International, IEEE, 2005, pp. 19-24.

Digital Library

[26]

G.W. Dong, C.H. Nie, B.W. Xu, L.L. Wang, An effective iterative metamorphic testing algorithm based on program path analysis, in: Quality Software, 2007. QSIC'07. Seventh International Conference on, IEEE, 2007, pp. 292-297.

Digital Library

[27]

L. Chen, L. Cai, J. Liu, Z. Liu, S. Wei, P. Liu, An optimized method for generating cases of metamorphic testing, in: Information Science and Service Science and Data Mining (ISSDM), 2012 6th International Conference on New Trends in, Taipei, Taiwan, China, 2012, pp. 439 - 443.

[28]

F.T. Chan, T.Y. Chen, I.K. Mak, Y.T. Yu, Proportional sampling strategy: guidelines for software testing practitioners, Information and Software Technology, 38 (1996) 775-782.

[29]

Z.Y. Zhang, W.K. Chan, T.H. Tse, P.F. Hu, Experimental study to compare the use of metamorphic testing and assertion checking, Journal of Software, 20 (2009) 2637- 2654.

[30]

T.Y. Chen, F.-C. Kuo, R. Merkel, On the statistical properties of testing effectiveness measures, Journal of Systems and Software, 79 (2006) 591-601.

Digital Library

[31]

R.A. DeMillo, R.J. Lipton, F.G. Sayward, Hints on test data selection: Help for the practicing programmer, Computer, 11 (1978) 34-41.

Digital Library

[32]

H. Zhu, P.A. Hall, J.H. May, Software unit test coverage and adequacy, ACM Computing Surveys (CSUR), 29 (1997) 366-427.

Digital Library

[33]

W.H. Press, Numerical recipes 3rd edition: The art of scientific computing, Cambridge university press, 2007.

Digital Library

[34]

G. Dong, Metamorphic testing techniques for error detection efficiency, in: School of Computer Science and Engineering, Southeast University, Nanjing, China, 2009.

[35]

M. Hutchins, H. Foster, T. Goradia, T. Ostrand, Experiments of the effectiveness of dataflow-and controlflow-based test adequacy criteria, in: Proceedings of the 16th international conference on Software engineering, IEEE Computer Society Press, 1994, pp. 191-200.

Digital Library

[36]

G. License, Gcov: Gnu coverage tool, in.

Cited By

Ying ZTowey DChen TZhou Z(2024)MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00158(1184-1193)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00158
Ying ZTowey DBellotti AChen TZhou Z(2024)SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domainsInformation and Software Technology10.1016/j.infsof.2024.107528(107528)Online publication date: Jul-2024
https://doi.org/10.1016/j.infsof.2024.107528
Ying ZBellotti ATowey DChen TZhou Z(2022)Using Metamorphic Relation Violation Regions to Support a Simulation Framework for the Process of Metamorphic Testing2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00274(1722-1727)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00274
Show More Cited By

Index Terms

MD-ART: a test case generation method without test oracle problem
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Feedback-Directed Metamorphic Testing
Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a ...
Fault detection effectiveness of source test case generation strategies for metamorphic testing
MET '18: Proceedings of the 3rd International Workshop on Metamorphic Testing

Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is ...
Combined Source Code Approach for Test Case Prioritization
ICISS '18: Proceedings of the 1st International Conference on Information Science and Systems

Regression testing is an activity in the software testing process to ensure the software is validated and verified after modification occurred on software. It is costly process procedure which has been expected to reach half cost of the software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SCTDCP 2016: Proceedings of the 1st International Workshop on Specification, Comprehension, Testing, and Debugging of Concurrent Programs

September 2016

34 pages

ISBN:9781450345101

DOI:10.1145/2975954

General Chairs:
Zijiang Yang,
Ting Liu,
Daniel Xiapu Luo

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

the NSF of Jiangsu Province, China

Conference

ASE'16

Sponsor:

SIGAI
SIGSOFT
IEEE-CS

ASE'16: ACM/IEEE International Conference on Automated Software Engineering

September 3, 2016

Singapore, Singapore

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
113
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ying ZTowey DChen TZhou Z(2024)MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00158(1184-1193)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00158
Ying ZTowey DBellotti AChen TZhou Z(2024)SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domainsInformation and Software Technology10.1016/j.infsof.2024.107528(107528)Online publication date: Jul-2024
https://doi.org/10.1016/j.infsof.2024.107528
Ying ZBellotti ATowey DChen TZhou Z(2022)Using Metamorphic Relation Violation Regions to Support a Simulation Framework for the Process of Metamorphic Testing2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00274(1722-1727)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00274
Huang RSun WXu YChen HTowey DXia X(2021)A Survey on Adaptive Random TestingIEEE Transactions on Software Engineering10.1109/TSE.2019.294292147:10(2052-2083)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TSE.2019.2942921
Hui ZWang XHuang SYang S(2021)MT-ART: A Test Case Generation Method Based on Adaptive Random Testing and Metamorphic RelationIEEE Transactions on Reliability10.1109/TR.2021.310638970:4(1397-1421)Online publication date: Dec-2021
https://doi.org/10.1109/TR.2021.3106389
Hui ZHuang SChua CChen T(2020)Semiautomated Metamorphic Testing Approach for Geographic Information Systems: An Empirical StudyIEEE Transactions on Reliability10.1109/TR.2019.293156169:2(657-673)Online publication date: Jun-2020
https://doi.org/10.1109/TR.2019.2931561

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents