Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3486608.3486904acmconferencesArticle/Chapter ViewAbstractPublication PagessplashConference Proceedingsconference-collections
research-article

New ideas: automated engineering of metamorphic testing environments for domain-specific languages

Published: 22 November 2021 Publication History

Abstract

Two crucial aspects for the trustworthy utilization of domain-specific languages (DSLs) are their semantic correctness, and proper testing support for their users. Testing is frequently used to verify correctness, but is often done informally -- which may yield unreliable results -- and requires substantial effort for creating suitable test cases and oracles.
To alleviate this situation, we propose an automated technique for building metamorphic testing environments for DSLs. Metamorphic testing identifies expected relationships between the outputs of two consecutive tests, reducing the effort in specifying oracles and creating test cases manually. This new ideas paper presents the overarching concepts, the architecture and a prototype implementation. We illustrate our proposal using a DSL to model and simulate data centres.

Supplementary Material

Auxiliary Archive (splashws21slemain-p18-p-archive.zip)
This is a presentation video of my talk at SLE 2021 on our paper accepted. Two crucial aspects for the trustworthy utilization of domain-specific languages (DSLs) are their semantic correctness, and proper testing support for their users. Testing is frequently used to verify correctness, but is often done informally -- which may yield unreliable results -- and requires substantial effort for creating suitable test cases and oracles. To alleviate this situation, we propose an automated technique for building metamorphic testing environments for DSLs. Metamorphic testing identifies expected relationships between the outputs of two consecutive tests, reducing the effort in specifying oracles and creating test cases manually. This new ideas paper presents the overarching concepts, the architecture and a prototype implementation. We illustrate our proposal using a DSL to model and simulate data centres.
Auxiliary Presentation Video (splashws21slemain-p18-p-video.mp4)
This is a presentation video of my talk at SLE 2021 on our paper accepted. Two crucial aspects for the trustworthy utilization of domain-specific languages (DSLs) are their semantic correctness, and proper testing support for their users. Testing is frequently used to verify correctness, but is often done informally -- which may yield unreliable results -- and requires substantial effort for creating suitable test cases and oracles. To alleviate this situation, we propose an automated technique for building metamorphic testing environments for DSLs. Metamorphic testing identifies expected relationships between the outputs of two consecutive tests, reducing the effort in specifying oracles and creating test cases manually. This new ideas paper presents the overarching concepts, the architecture and a prototype implementation. We illustrate our proposal using a DSL to model and simulate data centres.

References

[1]
M. Boussaa, O. Barais, G. Sunyé, and B. Baudry. 2020. Leveraging metamorphic testing to automatically detect inconsistencies in code generator families. Softw. Test. Verification Reliab., 30, 1 (2020).
[2]
E. Bousse, T. Degueule, D. Vojtisek, T. Mayerhofer, J. DeAntoni, and B. Combemale. 2016. Execution framework of the GEMOC studio (tool demo). In Proceedings of the 2016 ACM SIGPLAN International Conference on Software Language Engineering. ACM, 84–89.
[3]
E. Bousse, D. Leroy, B. Combemale, M. Wimmer, and B. Baudry. 2018. Omniscient debugging for executable DSLs. J. Syst. Softw., 137 (2018), 261–288.
[4]
M. Brambilla, J. Cabot, and M. Wimmer. 2017. Model-Driven Software Engineering in Practice, Second Edition. Morgan & Claypool Publishers.
[5]
C.-ai Sun, G. Wang, B. Mu, H. Liu, Z.S. Wang, and T. Y. Chen. 2011. Metamorphic testing for web services: Framework and a case study. In ICWS. IEEE, 283–290.
[6]
C.-ai Sun, G. Wang, Q. Wen, D. Towey, and T. Y. Chen. 2016. MT4WS: An automated metamorphic testing system for web services. International Journal of High Performance Computing and Networking, 9, 1/2 (2016), 104–115.
[7]
R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. De Rose, and R. Buyya. 2011. CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software - Practice and Experience, 41, 1 (2011), 23–50. issn:0038-0644
[8]
P. C. Cañizares, A. Núñez, J. de Lara, and L. Llana. 2020. MT-EA4Cloud: A methodology for testing and optimising energy-aware cloud systems. J. Syst. Softw., 163 (2020), 110522.
[9]
T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. HKUST-CS98-01.
[10]
T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Comput. Surv., 51, 1 (2018), 4:1–4:27.
[11]
B. Combemale, R. France, J.-M. Jézéquel, B. Rumpe, J. Steel, and D. Vojtisek. 2017. Engineering modeling languages. Turning domain knowledge into tools. Chapman and Hall/CRC.
[12]
K. Du, M. Jiang, Z. Ding, H. Huang, and T. Shu. 2019. Metamorphic testing in fault localization of model transformations. In SOFL+MSVL. Springer, 299–314.
[13]
P. Gómez-Abajo, E. Guerra, J. de Lara, and M. G. Merayo. 2021. Wodel-Test: A model-based framework for language independent mutation testing. Software and Systems Modelling, 20 (2021), 767–793.
[14]
A. Gotlieb and B. Botella. 2003. Automated metamorphic testing. In COMPAC. IEEE, 34–40.
[15]
R. I. Hadiwijaya and M. M. Liem. 2015. Metamorphic testing and DSL for test cases & checker generators. Olympiads in Informatics, 9 (2015), 75–88.
[16]
D. Harel and B. Rumpe. 2000. Modeling Languages: Syntax, Semantics and All That Stuff, Part I: The Basic Stuff. ISR.
[17]
X. He, X. Chen, S. Cai, Y. Zhang, and G. Huang. 2018. Testing bidirectional model transformation using metamorphic testing. Information and Software Technology, 104 (2018), 109–129.
[18]
D. Jackson. 2006. Software Abstractions - Logic, Language, and Analysis. MIT Press, London, England. See also http://alloy.mit.edu/
[19]
M. Jiang, T. Y. Chen, F.-C. Kuo, Z. Zhou, and Z. Ding. 2014. Testing model transformation programs using metamorphic testing. In SEKE. Knowledge Systems Institute Graduate School, 94–99.
[20]
L. C. L. Kats, R. Vermaas, and E. Visser. 2011. Integrated language definition testing: enabling test-driven language development. In OOPSLA. ACM, 139–154.
[21]
S. Kelly and J.-P. Tolvanen. 2008. Domain-Specific Modeling - Enabling Full Code Generation. Wiley. isbn:978-0-470-03666-2
[22]
F. Khorram, E. Bousse, J.-M. Mottu, and G. Sunyé. 2021. Adapting TDL to Provide Testing Support for Executable DSLs. Journal of Object Technology, 20, 3 (2021), June, 6:1–15. issn:1660-1769 The 17th European Conference on Modelling Foundations and Applications (ECMFA 2021).
[23]
M. Kuhlmann and M. Gogolla. 2012. From UML and OCL to relational logic and back. In MoDELS (LNCS, Vol. 7590). Springer, 415–431.
[24]
V. Le, M. Afshari, and Z. Su. 2014. Compiler validation via equivalence modulo inputs. In PLDI. ACM, 216–226.
[25]
X. Lin, M. Simon, and N. Niu. 2020. Exploratory metamorphic testing for scientific software. Computing in Science Engineering, 22, 2 (2020), 78–87.
[26]
Philip Makedonski, Gusztáv Adamis, Martti Käärik, Finn Kristoffersen, Michele Carignani, Andreas Ulrich, and Jens Grabowski. 2019. Test descriptions with ETSI TDL. Softw. Qual. J., 27, 2 (2019), 885–917.
[27]
S. Van Mierlo, Y. Van Tendeloo, and H. Vangheluwe. 2018. A generalized stepping semantics for model debugging. In MoDELS Workshops (CEUR Workshop Proceedings, Vol. 2245). CEUR-WS.org, 541–546.
[28]
A. Núñez, P. C. Cañizares, M. Núñez, and R. M. Hierons. 2021. TEA-Cloud: A formal framework for testing cloud computing systems. IEEE Transactions on Reliability, 70, 1 (2021), 261–284.
[29]
Object Management Group. 2014. UML 2.4 OCL Specification. http://www.omg.org/spec/OCL/.
[30]
Object Management Group. 2017. UML 2.5.1 Specification. https://www.omg.org/spec/UML/About-UML/.
[31]
D. Ratiu, M. Voelter, and D. Pavletic. 2018. Automated testing of DSL implementations - experiences from building mbeddr. Softw. Qual. J., 26, 4 (2018), 1483–1518.
[32]
S. Segura, G. Fraser, A. B. Sánchez, and A. Ruiz Cortés. 2016. A survey on metamorphic testing. IEEE Trans. Software Eng., 42, 9 (2016), 805–824.
[33]
S. Segura, J. A. Parejo, J. Troya, and A. Ruiz-Cortes. 2018. Metamorphic testing of RESTful web APIs. IEEE Trans. Software Eng., 44, 11 (2018), 1083–1099.
[34]
S. Segura, D. Towey, Z. Q. Zhou, and T. Y. Chen. 2020. Metamorphic testing: Testing the untestable. IEEE Softw., 37, 3 (2020), 46–53.
[35]
D. Steinberg, F. Budinsky, M. Paternostro, and E. Merks. 2008. EMF: Eclipse Modeling Framework, 2^nd Edition. Addison-Wesley Professional.
[36]
J. Troya, S. Segura, and A. Ruiz Cortés. 2018. Automated inference of likely metamorphic relations for model transformations. J. Syst. Softw., 136 (2018), 188–208.
[37]
M. Voelter. 2013. DSL Engineering - Designing, Implementing and Using Domain-Specific Languages. dslbook.org. isbn:978-1-4812-1858-0 http://www.dslbook.org
[38]
E. J. Weyuker. 1982. On testing non-testable programs. Comput. J., 25, 4 (1982), 465–470.
[39]
H. Wu, J. Gray, and M. Mernik. 2008. Grammar-driven generation of domain-specific language debuggers. Softw. Pract. Exp., 38, 10 (2008), 1073–1103.
[40]
H. Wu, J. G. Gray, and M. Mernik. 2009. Unit testing for domain-specific languages. In Domain-Specific Languages, IFIP TC 2 Working Conference, DSL (LNCS, Vol. 5658). Springer, 125–147.
[41]
X. Xie, J. W. K. Ho, C. Murphy, G. E. Kaiser, B. Xu, and T. Y. Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. J. Syst. Softw., 84, 4 (2011), 544–558.
[42]
2021. Xtext. http://www.eclipse.org/Xtext/ (last accessed in July 2021).
[43]
M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid. 2018. Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In ASE. ACM, 132–142.

Cited By

View all
  • (2023)Gotten: A Model-Driven Solution to Engineer Domain-specific Metamorphic Testing Environments2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)10.1109/MODELS-C59198.2023.00021(65-69)Online publication date: 1-Oct-2023
  • (2022)From Coverage Computation to Fault Localization: A Generic Framework for Domain-Specific LanguagesProceedings of the 15th ACM SIGPLAN International Conference on Software Language Engineering10.1145/3567512.3567532(235-248)Online publication date: 29-Nov-2022

Index Terms

  1. New ideas: automated engineering of metamorphic testing environments for domain-specific languages

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SLE 2021: Proceedings of the 14th ACM SIGPLAN International Conference on Software Language Engineering
    October 2021
    176 pages
    ISBN:9781450391115
    DOI:10.1145/3486608
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 November 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. DSLs
    2. Metamorphic Testing
    3. Model-Driven Engineering
    4. Sofware Language Engineering

    Qualifiers

    • Research-article

    Funding Sources

    • Spanish Ministry of Science
    • Madrid-UCM young doctors' program
    • Madrid region

    Conference

    SLE '21
    Sponsor:

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)13
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 09 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Gotten: A Model-Driven Solution to Engineer Domain-specific Metamorphic Testing Environments2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)10.1109/MODELS-C59198.2023.00021(65-69)Online publication date: 1-Oct-2023
    • (2022)From Coverage Computation to Fault Localization: A Generic Framework for Domain-Specific LanguagesProceedings of the 15th ACM SIGPLAN International Conference on Software Language Engineering10.1145/3567512.3567532(235-248)Online publication date: 29-Nov-2022

    View Options

    Get Access

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media