short-paper

Open access

QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit

Authors:

Daniel Fortunato,

Rui AbreuAuthors Info & Claims

ISSTA 2022: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 797 - 800

https://doi.org/10.1145/3533767.3543296

Published: 18 July 2022 Publication History

Abstract

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum gates, we describe our efforts in developing a tool, coined QMutPy, leveraging the well-known open-source mutation tool MutPy. We further discuss the design and implementation of QMutPy, and the usage of a novel set of mutation operators that generate mutants for qubit measurements and gates. To evaluate QMutPy’s performance, we conducted a preliminary study on 11 real quantum programs written in the IBM’s Qiskit library. QMutPy has proven to be an effective quantum mutation tool, providing insight into the current state of quantum tests. QMutPy is publicly available at https://github.com/danielfobooss/mutpy. Tool demo: https://youtu.be/fC4tOY5trqc.

References

[1]

Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz Machalica, Satish Chandra, and Erik Meijer. 2021. What It Would Take to Use Mutation Testing in Industry–A Study at Facebook. arxiv:2010.13464.

[2]

Austin Bingham. [n.d.]. Cosmic Ray: mutation testing for Python. https://github.com/sixty-north/cosmic-ray

[3]

Mark Fingerhuth, Tomáš Babej, and Peter Wittek. 2018. Open source software in quantum computing. PLOS ONE.

[4]

Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs Written in QISKit. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 358–359. https://doi.org/10.1109/ICSE-Companion55297.2022.9793776

[5]

Python Software Foundation. 2021. Python – pass statement. https://docs.python.org/3/tutorial/controlflow.html#pass-statements Accessed: 2021-08-24.

[6]

Konrad Hałas. 2011. MutPy: A Mutation Testing Tool for Python 3.x Source Code. https://github.com/mutpy/mutpy Accessed: 2021-01-18.

[7]

Hadi Hemmati, Andrea Arcuri, and Lionel Briand. 2013. Achieving Scalable Model-Based Testing through Test Case Diversity. 22, 1 (2013), Article 6, March, 42 pages. issn:1049-331X https://doi.org/10.1145/2430536.2430540

Digital Library

[8]

Shahin Honarvar, Mohammad Reza Mousavi, and Rajagopal Nagarajan. 2020. Property-Based Testing of Quantum Programs in Q#. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW’20). Association for Computing Machinery, New York, NY, USA. 430–435. isbn:9781450379632 https://doi.org/10.1145/3387940.3391459

Digital Library

[9]

Anders Hovmöller. 2016. Mutmut: a Python mutation testing system. https://github.com/boxed/mutmut Accessed: 2021-01-18.

[10]

Yue Jia and Mark Harman. 2010. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, 37, 5 (2010), 649–678.

[11]

Natalia Juristo, Ana M Moreno, and Wolfgang Strigel. 2006. Guest editors’ introduction: Software testing practices in industry. IEEE software, 23, 4 (2006), 19–21.

[12]

Rafaqut Kazmi, Dayang N. A. Jawawi, Radziah Mohamad, and Imran Ghani. 2017. Effective Regression Test Case Selection: A Systematic Literature Review. ACM Comput. Surv., 50, 2 (2017), Article 29, May, 32 pages. issn:0360-0300

[13]

Evan Kepner. [n.d.]. mutatest: Python mutation testing. https://github.com/EvanKepner/mutatest

[14]

Gushu Li, Li Zhou, Nengkun Yu, Yufei Ding, Mingsheng Ying, and Yuan Xie. 2020. Projection-Based Runtime Assertions for Testing and Debugging Quantum Programs. Proc. ACM Program. Lang., 4, OOPSLA (2020), Article 150, Nov., 29 pages. https://doi.org/10.1145/3428218

Digital Library

[15]

P. Liu, S. Hu, M. Pistoia, C. R. Chen, and J. M. Gambetta. 2019. Stochastic Optimization of Quantum Programs. Computer, 52, 6 (2019), 58–67.

[16]

Eñaut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A Mutation Analysis Tool for Quantum Software Testing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1266–1270. https://doi.org/10.1109/ASE51524.2021.9678563

Digital Library

[17]

Javier Pellejero. 2020. MTQC: Mutation Testing for Quantum Computing. https://javpelle.github.io/MTQC Accessed: 2021-01-18.

[18]

Goran Petrović and Marko Ivanković. 2018. State of Mutation Testing at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’18). Association for Computing Machinery, New York, NY, USA. 163–171. isbn:9781450356596 https://doi.org/10.1145/3183519.3183521

Digital Library

[19]

Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2021. Does Mutation Testing Improve Testing Practices? In Proc. of the 43rd IEEE/ACM ICSE.

Digital Library

[20]

Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2021. Practical Mutation Testing at Scale: A view from Google. IEEE TSE.

[21]

Peter W. Shor. 1999. Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer. SIAM Rev., 41, 2 (1999), 303–332. https://doi.org/10.1137/S0036144598347011

Digital Library

[22]

Jiyuan Wang, Ming Gao, Yu Jiang, Jianguang Lou, Yue Gao, Dongmei Zhang, and Jiaguang Sun. 2018. QuanFuzz: Fuzz Testing of Quantum Program. arxiv:1810.10310.

[23]

Jianjun Zhao. 2020. Quantum Software Engineering: Landscapes and Horizons. arxiv:2007.07047.

[24]

Pengzhan Zhao, Jianjun Zhao, and Lei Ma. 2021. Identifying Bug Patterns in Quantum Programs. In Proc. of the 2nd Q-SE.

Cited By

Virgínio TBastos LBezerra CRibeiro MMachado I(2024)How Aware Are We of Test Smells in Quantum Software Systems? A Preliminary Empirical EvaluationProceedings of the XXIII Brazilian Symposium on Software Quality10.1145/3701625.3701676(383-393)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3701625.3701676
Muqeet AAli SArcaini PFilkov VRay BZhou M(2024)Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum ComputersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695275(2130-2141)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695275
Shi JXiao ZShi HJiang YLi X(2024)QuanTest: Entanglement-Guided Testing of Quantum Neural Network SystemsACM Transactions on Software Engineering and Methodology10.1145/368884034:2(1-32)Online publication date: 19-Aug-2024
https://dl.acm.org/doi/10.1145/3688840
Show More Cited By

Index Terms

QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software maintenance tools

Recommendations

Mutation testing of quantum programs written in QISKit
ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

There is an inherent lack of knowledge and technology to test a quantum program properly. In this paper, building on the definition of syntactically equivalent quantum operations, we investigated a novel set of mutation operators to generate mutants ...
Quantum Software Testing 101
ICSE-Companion '24: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

Quantum software testing (QST) is an emerging research area within quantum software engineering (QSE) to ensure quantum software functional and non-functional correctness and dependability. Since quantum computers perform computations significantly ...
Gate Branch Coverage: A Metric for Quantum Software Testing
QSE-NE 2024: Proceedings of the 1st ACM International Workshop on Quantum Software Engineering: The Next Evolution

The inherent lack of technologies and knowledge from software developers about the intricacies of quantum physics constitutes a heavy hindrance in the development of correct quantum software. Therefore, quantum computing testing techniques are currently ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2022: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2022

808 pages

ISBN:9781450393799

DOI:10.1145/3533767

General Chair:
Sukyoung Ryu
KAIST, South Korea
,
Program Chair:
Yannis Smaragdakis
University of Athens, Greece

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ISSTA '22

Sponsor:

SIGSOFT

ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis

July 18 - 22, 2022

Virtual, South Korea

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
666
Total Downloads

Downloads (Last 12 months)267
Downloads (Last 6 weeks)30

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Virgínio TBastos LBezerra CRibeiro MMachado I(2024)How Aware Are We of Test Smells in Quantum Software Systems? A Preliminary Empirical EvaluationProceedings of the XXIII Brazilian Symposium on Software Quality10.1145/3701625.3701676(383-393)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3701625.3701676
Muqeet AAli SArcaini PFilkov VRay BZhou M(2024)Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum ComputersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695275(2130-2141)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695275
Shi JXiao ZShi HJiang YLi X(2024)QuanTest: Entanglement-Guided Testing of Quantum Neural Network SystemsACM Transactions on Software Engineering and Methodology10.1145/368884034:2(1-32)Online publication date: 19-Aug-2024
https://dl.acm.org/doi/10.1145/3688840
Li YPei HHuang LYin BCai K(2024)Automatic Repair of Quantum Programs via Unitary OperationACM Transactions on Software Engineering and Methodology10.1145/366460433:6(1-43)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664604
Fortunato DCampos JAbreu RKhan AAkbar MLiang P(2024)Gate Branch Coverage: A Metric for Quantum Software TestingProceedings of the 1st ACM International Workshop on Quantum Software Engineering: The Next Evolution10.1145/3663531.3664753(15-18)Online publication date: 16-Jul-2024
https://dl.acm.org/doi/10.1145/3663531.3664753
Paltenghi MPradel M(2024)Analyzing Quantum Programs with LintQ: A Static Analysis Framework for QiskitProceedings of the ACM on Software Engineering10.1145/36608021:FSE(2144-2166)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660802
Long PZhao J(2024)Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration TestingACM Transactions on Software Engineering and Methodology10.1145/365633933:6(1-61)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3656339
Juárez-Ramírez RNavarro CJiménez SRamírez ATapia-Ibarra VGuerra-García CPerez-Gonzalez HFernández-y-Fernández C(2024)A Taxonomic View of the Fundamental Concepts of Quantum Computing–A Software Engineering PerspectiveProgramming and Computing Software10.1134/S036176882308010849:8(682-704)Online publication date: 24-Jan-2024
https://dl.acm.org/doi/10.1134/S0361768823080108
Muqeet AYue TAli SArcaini P(2024)Mitigating Noise in Quantum Software Testing Using Machine LearningIEEE Transactions on Software Engineering10.1109/TSE.2024.346297450:11(2947-2961)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3462974
Huang LPei HLi YYin BCai K(2024)A Strategy of Dynamic Random Testing with Hybrid Distance Metrics for Quantum Programs2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00011(1-12)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS62785.2024.00011
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten