A systematic mapping on software testing for blockchains
Abstract
The purpose of this study is to identify and classify studies published on software testing techniques applied to blockchain systems. Previously published reviews in related areas have a narrow focus and/or do not follow a systematic review protocol. We conducted a systematic mapping based on an initial selection of 1025 studies. A rigorous selection process resulted in a final pool of 17 primary studies. These studies are categorized with respect to the employed testing methods, considered quality attributes, and functionality. We observe that most of the publications focus on testing functional correctness or security, whereas the testing of runtime performance attracts less attention. Existing approaches mostly employ fuzz testing or mutation testing. Search-based testing is usually combined with these techniques. The application of model-based testing is rare. The adaptability of fuzz testing and model-based testing techniques to changing blockchain platforms and languages remains a concern. On the other hand, performance and scalability issues are noted for search-based techniques and mutation testing. The use and integration of multiple testing techniques also stand out as a viable research direction.
References
[1]
Guo H and Yu X A survey on blockchain technology and its security Blockchain: Res. Appl. 2022 3 544-545
[2]
Chapman P, Xu D, Deng L, and Xiong Y Deviant: A mutation testing tool for solidity smart contracts IEEE Int. Conf. Blockchain 2019
[3]
Arsat N, Bakar NSAA, and Yahya N Testing in blockchain-based systems: a systematic review Int. Conf. Cyber IT Serv. Manag. (CITSM) 2022
[4]
Kitchenham B and Charters S Guidelines for Performing Systematic Literature Reviews in Software Engineering 2007 Keele University and Durham University
[5]
Benabbou, C., Gurcan, O.: A survey of verification, validation and testing solutions for smart contracts. In: 2021 Third International Conference on Blockchain Computing and Applications (BCCA), pp. 57–64 (2021).
[6]
Sujeetha, R., Deiva Preetha, C.A.S.: A literature survey on smart contract testing and analysis for smart contract based blockchain application development. In: 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), pp. 378–385 (2021).
[7]
Mei, X., Ashraf, I., Jiang, B., Chan, W.K.: A fuzz testing service for assuring smart contracts. In: 2019 IEEE 19th International Conference on Software Quality, Reliability and Security, pp. 544–545 (2019).
[8]
Liu, Y., Li, Y., Lin, S.-W., Yan, Q.: Modcon: A model-based testing platform for smart contracts. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2020, pp. 1601–1605. Association for Computing Machinery, New York, NY, USA (2020).
[9]
Smetanin S, Ometov A, Komarov M, Masek P, and Koucheryavy Y Blockchain evaluation approaches: state-of-the-art and future perspective Sensors 2020 20 12 3358
[10]
Tolmach P, Li Y, Lin S-W, Liu Y, and Li Z A survey of smart contract formal specification and verification ACM Comput. Surv. 2021
[11]
Garfatta, I., Klai, K., Gaaloul, W., Graiet, M.: A survey on formal verification for solidity smart contracts. In: Proceedings of the 2021 Australasian Computer Science Week Multiconference. ACSW ’21. Association for Computing Machinery, New York, NY, USA (2021).
[12]
Liu J and Liu Z A survey on security verification of blockchain smart contracts IEEE Access 2019 7 77894-77904
[13]
Krichen, M., Lahami, M., Al-Haija, Q.A.: Formal methods for the verification of smart contracts: a review. In: 2022 15th International Conference on Security of Information and Networks (SIN), pp. 01–08 (2022).
[14]
Ilgi, G.S., Kayali, D., Olawale, P., Demir Erdem, B., Dimililer, K., Kirsal-Ever, Y.: Formal verification for security technologies in the blockchain with artificial intelligence: A survey. In: 2022 Innovations in Intelligent Systems and Applications Conference (ASYU), pp. 1–6 (2022).
[15]
Murray, Y., Anisi, D.A.: Survey of formal verification methods for smart contracts on blockchain. In: 2019 10th IFIP International Conference on New Technologies, Mobility and Security (NTMS), pp. 1–6 (2019).
[16]
Imperius NP and Alahmar AD Systematic mapping of testing smart contracts for blockchain applications IEEE Access 2022 10 112845-112857
[17]
Martín-Martín A, Thelwall M, Orduna-Malea E, and López-Cózar ED Google scholar, microsoft academic, scopus, dimensions, web of science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations Scientometrics 2021 126 871-906
[18]
Yang L, Zhang H, Shen H, Huang X, Zhou X, Rong G, and Shao D Quality assessment in systematic literature reviews: a software engineering perspective Inform. Softw. Technol. 2021 130
[19]
Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–10 (2014)
[20]
Zhang, P., Yu, J., Ji, S.: Adf-ga: Data flow criterion based test case generation for ethereum smart contracts. In: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. ICSEW’20, pp. 754–761. Association for Computing Machinery, New York, NY, USA (2020).
[21]
Kim KB and Lee J Automated generation of test cases for smart contract security analyzers IEEE Access 2020 8 209377-209392
[22]
Ji, S., Zhu, S., Zhang, P., Dong, H.: Data flow reduction based test case generation for smart contracts. In: 2022 29th Asia-Pacific Software Engineering Conference (APSEC), pp. 149–158 (2022).
[23]
Li, Z., Wu, H., Xu, J., Wang, X., Zhang, L., Chen, Z.: Musc: A tool for mutation testing of ethereum smart contract. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1198–1201 (2019).
[24]
Ivanova Y and Khritankov A Regularmutator: a mutation testing tool for solidity smart contracts Procedia Comput. Sci. 2020 178 75-83
[25]
Akca, S., Rajan, A., Peng, C.: Solanalyser: A framework for analysing and testing smart contracts. In: 2019 26th Asia-Pacific Software Engineering Conference (APSEC), pp. 482–489 (2019).
[26]
Barboni M, Morichetta A, and Polini A Sumo: a mutation testing approach and tool for the ethereum blockchain J. Syst. Softw. 2022 193
[27]
Olsthoorn, M., Stallenberg, D., Van Deursen, A., Panichella, A.: Syntest-solidity: Automated test case generation and fuzzing for smart contracts. In: 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 202–206 (2022).
[28]
Andesta, E., Faghih, F., Fooladgar, M.: Testing smart contracts gets smarter. In: 2020 10th International Conference on Computer and Knowledge Engineering (ICCKE), pp. 405–412 (2020).
[29]
Fooladgar, M., Arefzadeh, A., Faghih, F.: Testsmart: A tool for automated generation of effective test cases for smart contracts. In: 2021 11th International Conference on Computer Engineering and Knowledge (ICCKE), pp. 476–481 (2021).
[30]
Zhou, T., Liu, K., Li, L., Liu, Z., Klein, J., Bissyandé, T.F.: Smartgift: Learning to generate practical inputs for testing smart contracts. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 23–34 (2021).
[31]
Walker, M., Dubey, A., Laszka, A., Schmidt, D.: Platibart: a platform for transactive iot blockchain applications with repeatable testing, pp. 17–22 (2017).
[32]
Ashraf I, Ma X, Jiang B, and Chan WK Gasfuzzer: fuzzing ethereum smart contract binaries to expose gas-oriented exception security vulnerabilities IEEE Access 2020 8 99552-99564
[33]
Nguyen, T.D., Pham, L.H., Sun, J., Lin, Y., Minh, Q.T.: sFuzz: an efficient adaptive fuzzer for solidity smart contracts. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 778–788 (2020)
[34]
Truffle Suite. https://trufflesuite.com/. Accessed 31 Oct 2023
[35]
Ganache. https://trufflesuite.com/ganache/. Accessed 31 Oct 2023
[36]
Luu, L., Chu, D.H., Olickel, H., Saxena, P., Hobor, A.: Making smart contracts smarter. In: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 254–269 (2016)
[37]
Silva RA, Senger de Souza SR, and Lopes de Souza PS A systematic review on search based mutation testing Inform. Softw. Technol. 2017 81 19-35
Index Terms
- A systematic mapping on software testing for blockchains
Index terms have been assigned to the content through auto-classification.
Recommendations
The experimental applications of search-based techniques for model-based testing
Graphical abstractDisplay Omitted HighlightsA systematic review of applications of search-based techniques for model-based testing is provided.Four taxonomies are proposed to classify the applications based on the purpose, problems, solutions and ...
A Systematic Literature Review on Test Case Prioritization in Combinatorial Testing
ICACS '21: Proceedings of the 5th International Conference on Algorithms, Computing and SystemsIn recent years, regression testing is widely used and performed in industries during the maintenance phase of software system development. The increases of software regression test cases resulted in a higher complexity of the software system, due to ...
A mapping study on testing non-testable systems
The terms "Oracle Problem" and "Non-testable system" interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in ...
Comments
Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 07 April 2024
Accepted: 05 March 2024
Revision received: 03 March 2024
Received: 15 November 2023
Author Tags
Qualifiers
- Review-article
Funding Sources
- Türkiye Bilimsel ve Teknolojik Araştırma Kurumu
- Özyeğin University
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 22 Jan 2025