Abstract
Recommender systems are commonly based on a multi-armed bandit model. This model should be carefully tested because it affects the users, but it is technically complicated because of the test oracle problem and the stochastical nature of multi-armed bandit algorithms. Metamorphic testing is a testing method for problems without test oracles. In this paper, we propose a novel approach that applies metamorphic testing to the verification of the requirements for stochastic models. We propose a stochastic metamorphic relation (SMR) which is a composition of a sampling procedure and a determination function. We propose several relations for multi-armed bandit models and algorithms. Then, we implement those relations and test algorithms. Our experiment demonstrates that the proposed method can identify errors and that stochasticity of relations is essential.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.: The nonstochastic multiarmed bandit problem. SIAM J. Comput. 32, 48–77 (2002)
Barr, E., Harman, M., McMinn, P., Shahbaz, M., Yoo, S.: The oracle problem in software testing: a survey. IEEE Trans. Software Eng. 41(5), 507–525 (2015). https://doi.org/10.1109/TSE.2014.2372785
Cavenaghi, E., Sottocornola, G., Stella, F., Zanker, M.: Non stationary multi-armed bandit: empirical evaluation of a new concept drift-aware algorithm. Entropy 23(3), 380 (2021)
Chen, T.Y., et al.: Metamorphic testing: a review of challenges and opportunities. ACM Comput. Surv. (CSUR) 51(1), 1–27 (2018). https://doi.org/10.1145/3143561
Fischer, G.: User modeling in human-computer interaction. User Model. User-Adap. Inter. 11, 65–86 (2001)
Iakusheva, S., Khritankov, A.: Composite metamorphic relations for integration testing. In: 2022 8th International Conference on Computer Technology Applications, May 12–14, Vienna, Austria (2022). https://doi.org/10.1145/3543712.3543725
Khritankov, A., Pershin, N., Ukhov, N., Ukhov., A.: MLDev: data science experiment automation and reproducibility software. In: Pozanenko, A., Stupnikov, S., Thalheim, B., Mendez, E., Kiselyova, N. (eds.) Data Analytics and Management in Data Intensive Domains. Communications in Computer and Information Science, vol. 1620, pp. 3–18. Springer, Cham (2021). https://doi.org/10.1007/978-3-031-12285-9_1
Khritankov, A., Pilkevich, A.: Existence conditions for hidden feedback loops in online recommender systems. In: Zhang, W., Zou, L., Maamar, Z., Chen, L. (eds.) Web Information Systems Engineering - WISE 2021. Lecture Notes in Computer Science(), vol. 13081, pp. 267–274. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-91560-5_19
Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Disc. 18, 140–181 (2009)
Mao, C., Yi, X., Chen, T. Y.: Metamorphic robustness testing for recommender systems: a case study. In: 2020 7th International Conference on Dependable Systems and Their Applications (DSA), pp. 331–336. IEEE (2020)
Matković, P., Tumbas, P.: A comparative overview of the evolution of software development models. Int. J. Ind. Eng. Manag. 1(4), 163 (2010)
Pesu, D., Zhou, Z. Q., Zhen, J., Towey, D.: A monte Carlo method for metamorphic testing of machine translation services. In: Proceedings of the 3rd International Workshop on Metamorphic Testing, pp. 38–45 (2018)
ur Rehman, F., Izurieta, C.: Statistical metamorphic testing of neural network based intrusion detection systems. In: 2021 IEEE International Conference on Cyber Security and Resilience (CSR), pp. 20–26. IEEE(2021)
Ricci, F., Rokach, L., Shapira, B., Kantor, P.B.: Recommender Systems Handbook, 1st edn. Springer-Verlag, Berlin (2010)
Russo, D.J., Roy, B.V., Kazerouni, A., Osband, I., Wen, Z.: A Tutorial on Thompson Sampling. Found. Trends R Mach. Learn. 11(1), 1–96 (2018)
Slivkins, A.: Introduction to multi-armed bandits. Found. Trends Mach. Learn. 12(1–2), 1–286 (2019). https://doi.org/10.1561/2200000068
Wang, J.C., Meyer, M.C.: Testing the monotonicity or convexity of a function using regression splines. Can. J. Stat. 39(1), 89–107 (2011)
Zhou, Z.Q., Tse, T.H., Witheridge, M.: Metamorphic robustness testing: exposing hidden defects in citation statistics and journal impact factors. IEEE Trans. Softw. Eng. 47(6), 1164–1183 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Iakusheva, S., Khritankov, A. (2024). Metamorphic Testing for Recommender Systems. In: Ignatov, D.I., et al. Analysis of Images, Social Networks and Texts. AIST 2023. Lecture Notes in Computer Science, vol 14486. Springer, Cham. https://doi.org/10.1007/978-3-031-54534-4_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-54534-4_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54533-7
Online ISBN: 978-3-031-54534-4
eBook Packages: Computer ScienceComputer Science (R0)