research-article

Contextual Understanding and Improvement of Metamorphic Testing in Scientific Software Development

Authors:

Upulee Kanewala,

Nan NiuAuthors Info & Claims

ESEM '21: Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Article No.: 28, Pages 1 - 6

https://doi.org/10.1145/3475716.3484188

Published: 11 October 2021 Publication History

Abstract

Background: Metamorphic testing emerges as a simple and effective approach for testing scientific software; yet, its adoption in actual scientific software projects is less studied.

Aims: In order for the practitioners to better adopt metamorphic testing in their projects, we set out to first gain a deep understanding about the current qualify assurance workflow, testing practices, and tools.

Method: We propose to integrate various empirical sources, including artifact analysis, stakeholder interviews, and gap analysis from the literature.

Results: Applying our approach to the Open Water Analytics Stormwater Management Model project helped to identify four new needs requiring continued and more research: (1) systematic and explicit formulation of metamorphic relations, (2) metamorphic testing examples specific to the scientific software, (3) correlating metamorphic testing with regression testing, and (4) integrating metamorphic testing with build tools like CMake and continuous integration tools like GitHub Actions.

Conclusions: Integrating different empirical sources is promising for establishing a contextual understanding of software engineering practices, and for action research, such as workflow refinements and tool interventions, to be carried out in a principled manner.

References

[1]

Özgür Akgün, Ian P. Gent, Christopher Jefferson, Ian Miguel, and Peter Nightingale. 2018. Metamorphic Testing of Constraint Solvers. In Proceedings of the International Conference on Principles and Practice of Constraint Programming (CP'20). Lille, France, 727--736.

[2]

Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (May 2015), 507--525.

Digital Library

[3]

Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. Comput. Surveys 51, 1 (April 2018), 4:1--4:27.

Digital Library

[4]

Tira Cohene and Steve Easterbrook. 2005. Contextual Risk Analysis for Interview Design. In Proceedings of the International Requirements Engineering Conference (RE'05). Paris, France, 95--104.

Digital Library

[5]

Junhua Ding and Xin-Hua Hu. 2017. Application of Metamorphic Testing Monitored by Test Adequacy in a Monte Carlo Simulation Program. Software Quality Journal 25, 3 (September 2017), 841--869.

Digital Library

[6]

Junhua Ding, Dongmei Zhang, and Xin-Hua Hu. 2016. An Application of Metamorphic Testing for Testing Scientific Software. In Proceedings of the International Workshop on Metamorphic Testing (MET'16). Austin, TX, USA, 37--43.

Digital Library

[7]

Paul F. Dubois. 2012. Testing Scientific Programs. Computing in Science and Engineering 14, 4 (July/August 2012), 69--73.

Digital Library

[8]

Bonnie Hardin and Upulee Kanewala. 2018. Using Semi-Supervised Learning for Predicting Metamorphic Relations. In Proceedings of the International Workshop on Metamorphic Testing (MET'18). Gothenburg, Sweden, 14--17.

Digital Library

[9]

Konrad Hinsen. 2015. The approximation tower in computational science: why testing scientific software is difficult. Computing in Science and Engineering 17, 4 (July/August 2015), 72--77.

Digital Library

[10]

Upulee Kanewala and James M. Bieman. 2014. Testing Scientific Software: A Systematic Literature Review. Information & Software Technology 56, 10 (October 2014), 1219--1232.

Digital Library

[11]

Upulee Kanewala, James M. Bieman, and Asa Ben-Hur. 2016. Predicting Metamorphic Relations for Testing Scientific Software: A Machine Learning Approach using Graph Kernels. Software Testing, Verification & Reliability 26, 3 (May 2016), 245--269.

Digital Library

[12]

Upulee Kanewala and Tsong Yueh Chen. 2019. Metamorphic Testing: A Simple Yet Effective Approach for Testing Scientific Software. Computing in Science and Engineering 21, 1 (January/February 2019), 66--72.

Digital Library

[13]

Upulee Kanewala, Anders Lundgren, and James M Bieman. 2016. Automated metamorphic testing of scientific software. In Software Engineering for Science. Chapman and Hall/CRC, 185--210.

[14]

Charu Khatwani, Xiaoyu Jin, Nan Niu, Amy Koshoffer, Linda Newman, and Juha Savolainen. 2017. Advancing Viewpoint Merging in Requirements Engineering: A Theoretical Replication and Explanatory Study. Requirements Engineering 22, 3 (September 2017), 317--338.

Digital Library

[15]

Konstantin Kreyman, David Lorge Parnas, and Sanzheng Qiao. 1999. Inspection Procedures for Critical Programs that Model Physical Phenomena. Technical Report. McMaster University, Hamilton, ON, Canada.

[16]

Xuanyi Lin, Michelle Simon, and Nan Niu. 2018. Hierarchical Metamorphic Relations for Testing Scientific Software. In Proceedings of the International Workshop on Software Engineering for Science (SE4Science'18). Gothenburg, Sweden, 1--8.

Digital Library

[17]

Xuanyi Lin, Michelle Simon, and Nan Niu. 2020. Exploratory Metamorphic Testing for Scientific Software. Computing in Science and Engineering 22, 2 (March/April 2020), 78--87.

[18]

Xuanyi Lin, Michelle Simon, and Nan Niu. 2021. Scientific Software Testing Goes Serverless: Creating and Invoking Metamorphic Functions. IEEE Software 38, 1 (January/February 2021), 61--67.

Digital Library

[19]

Anders Lundgren and Upulee Kanewala. 2016. Experiences of Testing Bioinformatics Programs for Detecting Subtle Faults. In Proceedings of the International Workshop on Software Engineering for Science (SE4Science'16). Austin, TX, USA, 16--22.

Digital Library

[20]

Greg Miller. 2006. A Scientist's Nightmare: Software Problem Leads to Five Retractions. Science 314, 5807 (December 2006), 1856--1857.

[21]

Nan Niu, Amy Koshoffer, Linda Newman, Charu Khatwani, Chatura Samarasinghe, and Juha Savolainen. 2016. Advancing Repeated Research in Requirements Engineering: A Theoretical Replication of Viewpoint Merging. In Proceedings of the International Requirements Engineering Conference (RE'16). Beijing, China, 186--195.

[22]

Megan M. Olsen and Mohammad S. Raunak. 2016. Metamorphic Validation for Agent-Based Simulation Models. In Proceedings of the Summer Computer Simulation Conference (SummerSim'16). Montreal, Canada, 33.

Digital Library

[23]

Megan M. Olsen and Mohammad S. Raunak. 2019. Increasing Validity of Simulation Models Through Metamorphic Testing. IEEE Transactions on Reliability 68, 1 (March 2019), 91--108.

[24]

Open Water Analytics. 2021. Open Water Analytics Stormwater Management Model (OWA SWMM). https://github.com/OpenWaterAnalytics/Stormwater-Management-Model Last accessed: August 2021.

[25]

Zedong Peng, Xuanyi Lin, and Nan Niu. 2020. Unit Tests of Scientific Software: A Study on SWMM. In Proceedings of the International Conference on Computational Science (ICCS'20). Amsterdam, The Netherlands, 413--427.

[26]

Zedong Peng, Xuanyi Lin, Michelle Simon, and Nan Niu. 2021. Unit and Regression Tests of Scientific Software: A Study on SWMM. Journal of Computational Science 53 (July 2021), 101347:1-101347:13.

[27]

Christian R. Prause, Jürgen Werner, Kay Hornig, Sascha Bosecker, and Marco Kuhrmann. 2017. Is 100% Test Coverage a Reasonable Requirement? Lessons Learned from a Space Software Project. In Proceedings of the International Conference on Product-Focused Software Process Improvement (PROFES'17). Innsbruck, Austria, 351--367.

[28]

Karishma Rahman and Upulee Kanewala. 2018. Predicting Metamorphic Relations for Matrix Calculation Programs. In Proceedings of the International Workshop on Metamorphic Testing (MET'18). Gothenburg, Sweden, 10--13.

Digital Library

[29]

Sergio Segura, Gordon Fraser, Ana B. Sánchez, and Antonio Ruiz Cortés. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9 (September 2016), 805--824.

[30]

Morteza Pourreza Shahri, Madhusudan Srinivasan, Gillian Reynolds, Diane Bimczok, Indika Kahanda, and Upulee Kanewala. 2019. Metamorphic Testing for Quality Assurance of Protein Function Prediction Tools. In Proceedings of the International Conference on Artificial Intelligence Testing (AITest'19). Newark, CA, USA, 140--148.

[31]

Madhusudan Srinivasan, Morteza Pourreza Shahri, Indika Kahanda, and Upulee Kanewala. 2018. Quality Assurance of Bioinformatics Software: A Case Study of Testing a Biomedical Text Processing Tool Using Metamorphic Testing. In Proceedings of the International Workshop on Metamorphic Testing (MET'18). Gothenburg, Sweden, 26--33.

Digital Library

[32]

United States Environmental Protection Agency, Office of Research and Development, Center for Environmental Solutions & Emergency Response, Water Infrastructure Division. 2021. EPA ORD Stormwater Management Model. https://github.com/USEPA/Stormwater-Management-Model Last accessed: August 2021.

[33]

Igor Wiese, Ivanilton Polato, and Gustavo Pinto. 2020. Naming the Pain in Developing Scientific Software. IEEE Software 37, 4 (July/August 2020), 75--82.

Digital Library

[34]

Shiyu Yan, Xiaohua Yang, Meng Li, Hua Liu, and Zhaohui Liu. 2016. Research of Testing for Scientific Computing Software in the Area of Nuclear Power Based on Metamorphic Testing. In Proceedings of the Pacific Basin Nuclear Conference (PBNC'16). Beijing, China, 501--512.

Cited By

Duque-Torres APfahl DKlammer CFischer S(2023)Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00109(905-912)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00109
Duque-Torres APfahl D(2023)Towards a Complete Metamorphic Testing Pipeline2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00081(606-610)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00081
Duque-Torres APfahl DKlammer CFischer S(2022)Using Source Code Metrics for Predicting Metamorphic Relations at Method Level2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00132(1147-1154)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00132

Index Terms

Contextual Understanding and Improvement of Metamorphic Testing in Scientific Software Development
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Software defect analysis
        Software testing and debugging
  2. Software organization and properties

Recommendations

Feedback-Directed Metamorphic Testing
Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a ...
Metamorphic Testing: A Review of Challenges and Opportunities

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and ...
An application of metamorphic testing for testing scientific software
MET '16: Proceedings of the 1st International Workshop on Metamorphic Testing

One of the grand challenges for adequately testing scientific software is due to the oracle problem. Metamorphic testing is the promise technique for addressing the problem through checking the satisfaction of a metamorphic relation that defines the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '21: Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

October 2021

368 pages

ISBN:9781450386654

DOI:10.1145/3475716

General Chair:
Filippo Lanubile

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ESEM '21

Sponsor:

SIGSOFT

ESEM '21: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

October 11 - 15, 2021

Bari, Italy

Acceptance Rates

ESEM '21 Paper Acceptance Rate 24 of 124 submissions, 19%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
170
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Duque-Torres APfahl DKlammer CFischer S(2023)Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00109(905-912)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00109
Duque-Torres APfahl D(2023)Towards a Complete Metamorphic Testing Pipeline2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00081(606-610)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00081
Duque-Torres APfahl DKlammer CFischer S(2022)Using Source Code Metrics for Predicting Metamorphic Relations at Method Level2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00132(1147-1154)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00132

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents