Efficient regression testing of distributed real-time reactive systems in the context of model-driven development
Pages 1565 - 1587
Abstract
Regression testing is indispensable, especially for real-time distributed systems to ensure that existing functionalities are not affected by changes. Despite recent advances, regression testing for distributed systems remains challenging and extremely costly. Existing techniques often require running a failing system several times before detecting a regression. As a result, conventional approaches that use re-execution without considering the inherent non-determinism of distributed systems, and providing no (or low) control over execution are inadequate in many ways. In this paper, we present MRegTest, a replay-based regression testing framework in the context of model-driven development to facilitate deterministic replay of traces for detecting regressions while offering sufficient control for the purpose of testing over the execution of the changed system. The experimental results show that compared to the traditional approaches that annotate traces with timestamps and variable values MRegTest detects almost all regressions while reducing the size of the trace significantly and incurring similar runtime overhead.
References
[1]
Elbaum S, Rothermel G, and Penix J Techniques for improving regression testing in continuous integration development environments, ser. FSE,: new York, NY, USA: Assoc Comput. Mach. 2014 2014 235-245
[2]
Gabrielova, E.: End-to-end regression testing for distributed systems, ser. Middleware ’17. New York, NY, USA: Association for Computing Machinery, p. 9-12 (2017)
[3]
Aumayr, D., Marr, S., Béra, C., Boix, E.G., Mössenböck, H.: Efficient and deterministic record & replay for actor languages, ser. ManLang ’18. New York, NY, USA: Association for Computing Machinery, (2018)
[4]
Tveito, L., Johnsen, E.B., Schlatte, R.: Global reproducibility through local control for distributed active objects, In FASE, ser. Lecture Notes in Computer Science, vol. 12076. Springer, pp. 140–160 (2020)
[5]
Lanese, I., Palacios, A., Vidal, G.: Causal-consistent replay debugging for message passing programs, In IFIP, ser Lecture Notes in Computer Science, Springer 11535, 167–184 (2019)
[6]
Kazuhiro Shibanai, K., Watanabe, T.: Distributed functional reactive programming on actor-based runtime, In SIGPLAN. ACM, pp. 13–22 (2018)
[7]
Sen, K., Kalasapur, S., Brutch, T., Gibbs, S.: Jalangi: a selective record-replay and dynamic analysis framework for JavaScript, ser. ESEC/FSE. New York, NY, USA: Association for Computing Machinery, p. 488-498 (2013)
[8]
Leshed, G., Haber, E.M., Matthews, T., Lau, T.: Coscripter: Automating & sharing how-to knowledge in the enterprise, In SIGCHI, ser. CHI ’08. New York, NY, USA: association for Computing Machinery, p. 1719-1728 (2008)
[9]
QF-Test website, https://www.qfs.de/en.html, retrieved (2021)
[10]
Holmes, A., Kellogg, M.: Automating functional tests using selenium, In AGILE’06, (2006)
[11]
Smartbear, “TestComplete,” https://smartbear.com/product/testcomplete/overview/, retrieved (2021)
[12]
Kresse, A., Kruse, P.M.: Development and maintenance efforts testing graphical user interfaces: a comparison, ser. A-TEST. New York, NY, USA: association for Computing Machinery, p. 52-58 (2016)
[13]
Sigelman BH, Barroso LA, Burrows M, Stephenson P, Plakal M, Beaver D, Jaspan S, and Shanbhag C Dapper, a Large-Scale Distributed Systems Tracing Infrastructure 2010 Tech. Rep. Google Inc
[14]
Park, S., Lu, S., Zhou, Y.: Ctrigger: Exposing atomicity violation bugs from their hiding places, ser. ASPLOS XIV. New York, NY, USA: association for Computing Machinery, p. 25-36 (2009)
[15]
Babaei, M., Dingel, J.: MRegTest repository, https://github.com/MajidGitHubRepos/MRegTest, retrieved (2021)
[16]
Babaei, M., Bagherzadeh, M., Dingel, J.: Efficient reordering and replay of execution traces of distributed reactive systems in the context of model-driven development, In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, ser. MODELS’20. New York, NY, USA: Association for Computing Machinery, p. 285-296 (2020)
[17]
Babaei, M.: Regression testing of distributed real-time embedded systems in the context of model-driven development, PhD dissertation, Kingston, Ontario, Canada, (2021)
[18]
Ural H and Whittier D Distributed testing without encountering controllability and observability problems Inf. Process. Lett. 2003 88 3 133-141
[19]
Hierons RM, Merayo MG, and Nunez M Wotawa F, Nica M, and Kushik N Controllability through nondeterminism in distributed testing Testing Software and Systems 2016 Cham Springer International Publishing 89-105
[20]
Cacciari L and Rafiq O Controllability and observability in distributed testing Inf. Softw. Technol. 1999 41 11 767-780
[21]
Terragni, V., Cheung, S., Zhang, C.: Recontest: effective regression testing of concurrent programs,’ In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, 1, pp. 246–256 (2015)
[22]
Yu, T., Srisa-an, W., Rothermel, G.: SimRT: An automated framework to support regression testing for data races, ser. ICSE. New York, NY, USA: association for Computing Machinery, p. 48-59 (2014)
[23]
Joshi, P., Ganai, M., Balakrishnan, G., Gupta, A., Papakonstantinou, N.:“SETSUDundefined: Perturbation-based testing framework for scalable distributed systems, ser. TRIOS. New York, NY, USA: association for Computing Machinery, (2013)
[24]
Lima, B.: Automated scenario-based integration testing of time-constrained distributed systems, In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pp. 486–488 (2019)
[25]
Bianchi FA, Margara A, and Pezzè M A survey of recent trends in testing concurrent software systems IEEE Trans. Software Eng. 2018 44 8 747-783
[26]
Torens, C., Ebrecht, L.: RemoteTest: a framework for testing distributed systems, pp. 441–446 (2010)
[27]
Milka, G., Rzadca, K.: Dfuntest: a testing framework for distributed applications. CoRR, (2018)
[28]
Foundation, E.: Eclipse Papyrus for real time (Papyrus-RT), https://www.eclipse.org/papyrus-rt, 2016, retrieved June 5, (2019)
[29]
Jahed, Karim: Papyrus-RT Distribution, https://github.com/kjahed/papyrusrt-distribution, retrieved: 2019-8-10 (2019)
[30]
Babaei, M., Bagherzadeh, M., Dingel, J.: Mreplayer: a trace replayer of distributed UML-RT models,” ser. MODELS’20. New York, NY, USA: association for Computing Machinery, (2020)
[31]
The Object Management Group, UML Superstructure Specification. Version 2.5.1, http://www.omg.org/spec/UML/2.5.1/PDF, 2017, retrieved August 23, (2019)
[32]
Stivan, G., Peruffo, A., Haller, P.: Akka.js: towards a portable actor runtime environment, ser. AGERE. New York, NY, USA: Association for Computing Machinery, p. 57-64 (2015)
[33]
Bykov, S., Geller, A., Kliot, G., Larus, J.R., Pandya, R., Thelin, J.: Orleans: Cloud computing for everyone, ser. SOCC. New York, NY, USA: Association for Computing Machinery, (2011)
[34]
McCaffrey, C.: Building the Halo 4 services with Orleans, 2015, presentation at QCon London
[35]
Boer, F.D., Serbanescu, V., Hähnle, R., Henrio, L., Rochas, J., Din, C.C., Johnsen, E.B., Sirjani, M., Khamespanah, E., Fernandez-Reyes, K., Yang, A.M.: A survey of active object languages,” ACM Comput. Surv., 50(5), (2017)
[36]
Rothermel G and Harrold MJ Analyzing regression test selection techniques IEEE Trans. Software Eng. 1996 22 8 529-551
[37]
Hammoudi, M.: Regression testing of web applications using record/replay tools, ser. FSE. New York, NY, USA: Association for Computing Machinery, p. 1079-1081 (2016)
[38]
Selic, B., Gullekson, G., Ward, P.T.: Real-Time Object-Oriented Modeling. John Wiley and Sons New York, 2, (1994)
[39]
Posse E and Dingel J An executable formal semantics for UML-RT Softw. Syst. Model. 2016 15 1 179-217
[40]
IBM, IBM RSARTE, https://www.ibm.com/developerworks/community/wikis, 2016, retrieved July 19, (2016)
[41]
HCL, RealTime Software Tooling (RTist), https://www.hcltech.com/software/rtist, retrieved (2020)
[42]
Eclipse Foundation, Eclipse eTrice Real-Time Modeling Tools, https://www.eclipse.org/etrice, (2020)
[43]
Selic, B.: Using UML for modeling complex real-time systems, in Languages, compilers, and tools for embedded systems. Springer, pp. 250–260 (1998)
[44]
Bagherzadeh, M.: Model-level debugging in the context of the model driven development, PhD dissertation, Kingston, Ontario, Canada, (2019)
[45]
von der Beeck, M.: A formal semantics of UML-RT, In Model Driven Engineering Languages and Systems, O. Nierstrasz, J. Whittle, D. Harel, and G. Reggio, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 768–782 (2006)
[46]
Mealy GH A method for synthesizing sequential circuits Bell Syst. Tech. J. 1955 34 5 1045-1079
[47]
Ahmadi, R.: Testing models of reactive systems, PhD dissertation, Kingston, Ontario, Canada, (2019)
[48]
Esmaeilsabzali S, Day N, Atlee J, and Niu J Deconstructing the semantics of big-step modelling languages Requir. Eng. 2010 15 06 235-265
[49]
Parr, T.: Antlr (another tool for language recognition), https://www.antlr.org/, 2017, retrieved June 5, (2019)
[50]
Milner R A Calculus of Communicating Systems 1982 Berlin, Heidelberg Springer-Verlag
[51]
Hoare, C.A.R.: Communicating sequential processes, 21(8), 666-677, (1978) [Online]. Available: https://doi.org/10.1145/359576.359585
[52]
Kolovos, D.S., Paige, R.F., Polack, F.A.: The Epsilon transformation language, In International Conference on Theory and Practice of Model Transformations. Springer, pp. 46–60 (2008)
[53]
Dunfield, J.: Lecture notes in cisc 327 - software quality assurance, Fall (2018)
[54]
Rothermel G and Harrold M Analyzing regression test selection techniques IEEE Trans. Softw. Eng. 1996 22 8 529-551
[55]
Briand L, Labiche Y, and He S Automating regression test selection based on UML designs Inf. Softw. Technol. 2009 51 1 16-30
[56]
Balasubramanian, J., Tambe, S., Lu, C., Gokhale, A., Gill, C., Schmidt, D.C.: Adaptive failover for real-time middleware with passive replication, In 15th IEEE Symposium on Real-Time and Embedded Technology and Applications. IEEE, pp. 118–127 (2009)
[57]
Kahani, N., Hili, N., Cordy, J.R., Dingel, J.: Evaluation of UML-RT and Papyrus-RT for modelling self-adaptive systems, In Proceedings of the 9th International Workshop on Modelling in Software Engineering. IEEE Press, pp. 12–18 (2017)
[58]
Swartout W and Balzer R On the inevitable intertwining of specification and implementation Commun. ACM 1982 25 7 438-440
[59]
Magee, J., Kramer, J.: State Models and Java Programs. Wiley, (1999)
[60]
Bagherzadeh, M., Hili, N., Dingel, J.: Model-level, platform-independent debugging in the context of the model-driven development of real-time systems, In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017. New York, NY, USA: ACM, pp. 419–430 (2017)
[61]
“ATM UML-RT model,” https://github.com/MajidGitHubRepos/atm_umlrt, retrieved (2021)
[62]
Ekelund ED and Engström E Efficient regression testing based on test history: An industrial evaluation IEEE Int. Conf. Softw. Maint. Evol. (ICSME) 2015 2015 449-457
[63]
Yu T, Huang Z, and Wang C ConTesa: directed test suite augmentation for concurrent software IEEE Trans. Software Eng. 2020 46 4 405-419
[64]
Chen Y, Zhang S, Guo Q, Li L, Wu R, and Chen T Deterministic replay: a survey ACM Comput. Surv. 2015 48 2 171-1747
[65]
Hammoudi, M., Rothermel, G., Stocco, A.: WATERFALL: an incremental approach for repairing record-replay tests of web applications, ser. FSE. New York, NY, USA: Association for Computing Machinery, p. 751-762 (2016)
[66]
Pal D and Vain J A systematic approach on modeling refinement and regression testing of real-time distributed systems IFAC 2019 52 13 1091-1096
[67]
Yoo S and Harman M Regression testing minimization, selection and prioritization: A survey Softw. Test. Verif. Reliab. 2012 22 2 67-120
[68]
Biswas S, Mall R, Satpathy M, and Sukumaran S Regression test selection techniques: a survey Informatica (Slovenia) 2011 35 3 289-321
[69]
Engström, E., Runeson, P., Skoglund, M.: A systematic review on regression test selection techniques, 52(1), 14-30 (2010)
[70]
Orso A, Shi N, and Harrold MJ Scaling regression testing to large software systems SIGSOFT Softw. Eng. Notes 2004 29 6 241-251
[71]
Zhang, L., Kim, M., Khurshid, S.: Localizing failure-inducing program edits based on spectrum information, In 2011 27th IEEE International Conference on Software Maintenance (ICSM), pp. 23–32 (2011)
[72]
Gligoric, M., Eloussi, L., Marinov, D.: Practical regression test selection with dynamic file dependencies, ser. ISSTA,: New York, NY, USA: Association for Computing. Machinery 2015, 211–222 (2015)
[73]
Legunsen, O., Hariri, F., Shi, A., Lu, Y., Zhang, L., Marinov, D.: An extensive study of static regression test selection in modern software evolution, ser. FSE. New York, NY, USA: Association for Computing Machinery, p. 583-594 (2016)
[74]
Tools, G.E.: Build in the cloud: How the build system works, http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html, 2021, retrieved Jan 30, (2021)
[75]
Zhang, L.: Hybrid regression test selection, In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pp. 199–209 (2018)
[76]
Harrold, M.J., Jones, J. A., Li, T., Liang, D., Orso, A., Pennings, M., Sinha, S., Spoon, S. A., Gujarathi, A.: Regression test selection for java software, In Proceedings of the 16th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, ser. OOPSLA ’01. New York, NY, USA: Association for Computing Machinery, p. 312-326 (2001)
[77]
Orso, A., Shi, N., Harrold, M. J.: Scaling regression testing to large software systems, ser. SIGSOFT’04. New York, NY, USA: Association for Computing Machinery, p. 241-251 (2004)
[78]
Legunsen, O., Shi, A., Marinov, D.: STARTS:STAtic regression test selection, pp. 949–954 (2017)
[79]
Almasri, N., Tahat, L., Korel, B.: Automatically quantifying the impact of a change in systems (journal-first abstract), ser. ASE. New York, NY, USA: Association for Computing Machinery, p. 952 (2018)
[80]
Chen, Y., Probert, R. L., Ural, H.: Model-based regression test suite generation using dependence analysis, ser. A-MOST ’07. New York, NY, USA: Association for Computing Machinery, p. 54-62 (2007)
[81]
Biswas S, Mall R, Satpathy M, and Sukumaran S A model-based regression test selection approach for embedded applications SIGSOFT Softw. Eng. Notes 2009 34 4 1-9
[82]
Panigrahi, C. R., Mall, R.: Model-based regression test case prioritization, In Information Systems, Technology and Management, S. K. Prasad, H. M. Vin, S. Sahni, M. P. Jaiswal, and B. Thipakorn, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 380–385 (2010)
[83]
Honfi D, Molnár G, Micskei Z, and Majzik I Csöndes T, Kovács G, and Réthy G Model-based regression testing of autonomous robots SDL 2017: Model-Driven Engineering for Future Internet 2017 Cham Springer International Publishing 119-135
[84]
Biswas, S., Mall, R., Satpathy, M.: A regression test selection technique for embedded software, ACM Trans. Embed. Comput. Syst., 13(3), (2013)
[85]
Zech P, Kalb P, Felderer M, Atkinson C, and Breu R Model-based regression testing by OCL Int. J. Softw. Tools Technol. Transf. 2017 19 1 115-131
[86]
Zech, P., Felderer, M., Kalb, P., Breu, R.: A generic platform for model-based regression testing, In Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change, T. Margaria and B. Steffen, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 112–126 (2012)
[87]
Pal, D., Vain, J.: Model based approach for testing: distributed real-time systems augmented with online monitors. In: Lupeikiene, A., Vasilecas, O., Dzemyda, G. (eds.) Databases and Information Systems, pp. 142–157. Springer International Publishing, Cham (2018)
[88]
Korel, B., Tahat, L. H., Vaysburg, B.: Model based regression test reduction using dependence analysis, in International Conference on Software Maintenance, 2002. Proceedings., pp. 214–223 (2002)
[89]
Chen, Y., Probert, R. L., Ural, H.: Regression test suite reduction using extended dependence analysis, ser. SOQUA’07. New York, NY, USA: Association for Computing Machinery, p. 62-69 (2007)
[90]
Wu, Ye, Offutt, J.: Maintaining evolving component-based software with UML, In Seventh European Conference on Software Maintenance and Re-engineering, 2003. Proceedings., pp. 133–142 (2003)
[91]
Briand, L., Labiche, Y., Soccar, G.: Automating impact analysis and regression test selection based on UML designs, In International Conference on Software Maintenance, 2002. Proceedings., pp. 252–261 (2002)
[92]
Farooq, Q.-u.-a., Iqbal, M.Z.Z., Malik, Z. I., Nadeem, A.: An approach for selective state machine based regression testing, ser. A-MOST’07. New York, NY, USA: Association for Computing Machinery, p. 44-52 (2007)
[93]
Traon Y, Jeron T, Jezequel J, and Morel P Efficient object-oriented integration and regression testing IEEE Trans. Reliab. 2000 49 1 12-25
Recommendations
Efficient reordering and replay of execution traces of distributed reactive systems in the context of model-driven development
MODELS '20: Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and SystemsOrdering and replaying of execution traces of distributed systems is a challenging problem. State-of-the-art approaches annotate the traces with logical or physical timestamps. However, both kinds of timestamps have their drawbacks, including increased ...
Model-level, platform-independent debugging in the context of the model-driven development of real-time systems
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software EngineeringProviding proper support for debugging models at model-level is one of the main barriers to a broader adoption of Model Driven Development (MDD). In this paper, we focus on the use of MDD for the development of real-time embedded systems (RTE). We ...
Dependent-test-aware regression testing techniques
ISSTA 2020: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and AnalysisDevelopers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 06 March 2023
Accepted: 20 December 2022
Revision received: 13 December 2022
Received: 14 March 2022
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in