research-article

Public Access

Learning to sample: exploiting similarities across environments to learn performance models for configurable systems

Authors:

Pooyan Jamshidi,

Christian Kästner,

Norbert SiegmundAuthors Info & Claims

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 71 - 82

https://doi.org/10.1145/3236024.3236074

Published: 26 October 2018 Publication History

Abstract

Most software systems provide options that allow users to tailor the system in terms of functionality and qualities. The increased flexibility raises challenges for understanding the configuration space and the effects of options and their interactions on performance and other non-functional properties. To identify how options and interactions affect the performance of a system, several sampling and learning strategies have been recently proposed. However, existing approaches usually assume a fixed environment (hardware, workload, software release) such that learning has to be repeated once the environment changes. Repeating learning and measurement for each environment is expensive and often practically infeasible. Instead, we pursue a strategy that transfers knowledge across environments but sidesteps heavyweight and expensive transfer-learning strategies. Based on empirical insights about common relationships regarding (i) influential options, (ii) their interactions, and (iii) their performance distributions, our approach, L2S (Learning to Sample), selects better samples in the target environment based on information from the source environment. It progressively shrinks and adaptively concentrates on interesting regions of the configuration space. With both synthetic benchmarks and several real systems, we demonstrate that L2S outperforms state of the art performance learning and transfer-learning approaches in terms of measurement effort and learning accuracy.

References

[1]

Apache Storm. 2018. http://storm.apache.org/.

[2]

J. P. S. Alcocer, A. Bergel, S. Ducasse, and M. Denker. Performance evolution blueprint: Understanding the impact of software evolution on performance. In Proc. of Working Conference on Software Visualization (VISSOFT), pages 1–9. IEEE, 2013.

[3]

M. Artač, editor. Deliverable 5.2: DICE delivery tools-Intermediate version. 2017. http://www.dice-h2020.eu/.

[4]

A. Biedenkapp, M. T. Lindauer, K. Eggensperger, F. Hutter, C. Fawcett, and H. H. Hoos. Efficient parameter importance analysis via ablation with surrogates. In AAAI, pages 773–779, 2017.

[5]

C. M. Bishop. Pattern recognition and machine learning. Springer, New York, 2006.

Digital Library

[6]

B. Bodin, L. Nardi, M. Z. Zia, H. Wagstaff, G. Sreekar Shenoy, M. Emani, J. Mawer, C. Kotselidis, A. Nisbet, M. Lujan, B. Franke, P. H. Kelly, and M. O’Boyle. Integrating algorithmic parameters into benchmarking and design space exploration in 3D scene understanding. In Proceedings of the International Conference on Parallel Architectures and Compilation (PACT), pages 57–69. ACM, 2016.

Digital Library

[7]

A. Brunnert, A. van Hoorn, F. Willnecker, A. Danciu, W. Hasselbring, C. Heger, N. Herbst, P. Jamshidi, R. Jung, J. von Kistowski, A. Koziolek, J. Kross, S. Spinner, C. Vögele, J. Walter, and A. Wert. Performance-oriented devops: A research agenda. SPEC-RG-2015-01, RG DevOps Performance, 2015.

[8]

H. Chen, W. Zhang, and G. Jiang. Experience transfer for the configuration tuning in large-scale computing systems. IEEE Trans. on Knowledge and Data Eng. (TKDE), 23(3):388–401, 2011.

Digital Library

[9]

Y. Chen, E. Keogh, B. Hu, N. Begum, A. Bagnall, A. Mueen, and G. Batista. The ucr time series classification archive, July 2015.

[10]

www.cs.ucr.edu/~eamonn/time_ series_data/.

[11]

J. Cito, P. Leitner, T. Fritz, and H. C. Gall. The making of cloud applications: An empirical study on software development for the cloud. In Proc. Int’l Symp. Foundations of Software Engineering (FSE), pages 393–403. ACM, 2015.

Digital Library

[12]

T. M. Cover and J. A. Thomas. Elements of information theory. John Wiley & Sons, 2012.

[13]

A. Elkhodary, N. Esfahani, and S. Malek. Fusion: A framework for engineering self-tuning self-adaptive software systems. In Proc. Int’l Symp. Foundations of Software Engineering (FSE), pages 7–16. ACM, 2010.

Digital Library

[14]

N. Esfahani, A. Elkhodary, and S. Malek. A learning-based framework for engineering feature-oriented self-adaptive software systems. IEEE Trans. Softw. Eng. (TSE), 39(11):1467–1493, 2013.

Digital Library

[15]

A. Filieri, H. Hoffmann, and M. Maggio. Automated multi-objective control for self-adaptive software design. In Proc. Int’l Symp. Foundations of Software Engineering (FSE), pages 13–24. ACM, 2015.

Digital Library

[16]

K. C. Foo, Z. M. J. Jiang, B. Adams, A. E. Hassan, Y. Zou, and P. Flora. An industrial case study on the automated detection of performance regressions in heterogeneous environments. In Proc. Int’l Conf. Software Engineering (ICSE), pages 159–168. IEEE Press, 2015.

Digital Library

[17]

J. Friedman, T. Hastie, and R. Tibshirani. The elements of statistical learning, volume 1. Springer series in statistics New York, 2001.

[18]

J. H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232, 2001.

[19]

A. Grebhahn, N. Siegmund, H. Köstler, and S. Apel. Performance prediction of multigrid-solver configurations. In Software for Exascale Computing-SPPEXA 2013-2015, pages 69–88. Springer, 2016.

[20]

J. Guo, K. Czarnecki, S. Apel, N. Siegmund, and A. Wasowski. Variability-aware performance prediction: A statistical learning approach. In Proc. Int’l Conf. Automated Software Engineering (ASE), pages 301–311. IEEE, 2013.

Digital Library

[21]

M. Harchol-Balter. Performance modeling and design of computer systems: queueing theory in action. Cambridge University Press, 2013.

Digital Library

[22]

C. Henard, M. Papadakis, M. Harman, and Y. Le Traon. Combining multi-objective search and constraint solving for configuring large software product lines. In Proc. Int’l Conf. Software Engineering (ICSE), pages 517–528. IEEE, 2015.

Digital Library

[23]

R. R. Hocking. A biometrics invited paper. the analysis and selection of variables in linear regression. Biometrics, 32(1):1–49, 1976.

[24]

H. Hoffmann, S. Sidiroglou, M. Carbin, S. Misailovic, A. Agarwal, and M. Rinard. Dynamic knobs for responsive power-aware computing. In In Proc. of Int’l Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2011.

Digital Library

[25]

H. H. Hoos. Automated algorithm configuration and parameter tuning. In Autonomous search, pages 37–71. Springer, 2011.

[26]

H. H. Hoos. Programming by optimization. Communications of the ACM, 55(2):70– 80, 2012.

Digital Library

[27]

K. Hoste, A. Phansalkar, L. Eeckhout, A. Georges, L. K. John, and K. De Bosschere. Performance prediction based on inherent program similarity. In Proc. of the International Conference on Parallel Architectures and Compilation Techniques (PACT), pages 114–122. ACM, 2006.

Digital Library

[28]

R. J. Hyndman and A. B. Koehler. Another look at measures of forecast accuracy. International journal of forecasting, 22(4):679–688, 2006.

[29]

P. Jamshidi and G. Casale. An uncertainty-aware approach to optimal configuration of stream processing systems. In Proc. Int’l Symp. on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 39–48. IEEE, September 2016.

[30]

P. Jamshidi, M. Ghafari, A. Ahmad, and C. Pahl. A framework for classifying and comparing architecture-centric software evolution research. In Proc. of European Conference on Software Maintenance and Reengineering (CSMR), pages 305–314. IEEE, 2013.

Digital Library

[31]

P. Jamshidi, N. Siegmund, M. Velez, C. Kästner, A. Patel, and Y. Agarwal. Transfer learning for performance modeling of configurable systems: An exploratory analysis. In Proc. Int’l Conf. Automated Software Engineering (ASE). ACM, 2017.

Digital Library

[32]

P. Jamshidi, M. Velez, C. Kästner, N. Siegmund, and P. Kawthekar. Transfer learning for improving model predictions in highly configurable software. In Proc. Int’l Symp. Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE, 2017.

Digital Library

[33]

P. Kawthekar and C. Kästner. Sensitivity analysis for building evolving and & adaptive robotic software. In Proceedings of the IJCAI Workshop on Autonomous Mobile Service Robots (WSR), 7 2016.

[34]

D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[35]

E. Kocaguneli, T. Menzies, and E. Mendes. Transfer learning in effort estimation. Empirical Software Engineering, 20(3):813–843, 2015.

Digital Library

[36]

R. Krishna, T. Menzies, and W. Fu. Too much automation? The bellwether effect and its implications for transfer learning. In Proc. Int’l Conf. Automated Software Engineering (ASE), pages 122–131. ACM, 2016.

Digital Library

[37]

D. R. Kuhn, R. N. Kacker, and Y. Lei. Introduction to combinatorial testing. CRC press, 2013.

Digital Library

[38]

P. Leitner and J. Cito. Patterns in the chaos - a study of performance variation and predictability in public IaaS clouds. ACM Trans. on Internet Technology (TOIT), 16(3):15, 2016.

Digital Library

[39]

D. C. Montgomery. Design and analysis of experiments. John Wiley & Sons, 2017.

[40]

A. Murashkin, M. Antkiewicz, D. Rayside, and K. Czarnecki. Visualization and exploration of optimal variants in product line engineering. In Proc. Int’l Software Product Line Conference (SPLC), pages 111–115. ACM, 2013.

Digital Library

[41]

V. Nair, T. Menzies, N. Siegmund, and S. Apel. Faster discovery of faster system configurations with spectral learning. arXiv preprint arXiv:1701.08106, 2017.

[42]

V. Nair, T. Menzies, N. Siegmund, and S. Apel. Faster discovery of faster system configurations with spectral learning. Automated Software Engineering, pages 1–31, 2017.

Digital Library

[43]

V. Nair, T. Menzies, N. Siegmund, and S. Apel. Using bad learners to find good configurations. In Proc. Int’l Symp. Foundations of Software Engineering (FSE), ESEC/FSE 2017, pages 257–267, New York, NY, USA, 2017. ACM.

Digital Library

[44]

J. Nam and S. Kim. Heterogeneous defect prediction. In Proc. Int’l Symp. Foundations of Software Engineering (FSE), pages 508–519. ACM, 2015.

Digital Library

[45]

J. Nam, S. J. Pan, and S. Kim. Transfer defect learning. In Proc. Int’l Conf. Software Engineering (ICSE), pages 382–391. IEEE, 2013.

Digital Library

[46]

J. Oh, D. Batory, M. Myers, and N. Siegmund. Finding product line configurations with high performance by random sampling. In Proc. Int’l Symp. Foundations of Software Engineering (FSE). ACM, 2017.

Digital Library

[47]

R. Olaechea, D. Rayside, J. Guo, and K. Czarnecki. Comparison of exact and approximate multi-objective optimization for software product lines. In Proc. Int’l Software Product Line Conference (SPLC), pages 92–101. ACM, 2014.

Digital Library

[48]

T. Osogami and S. Kato. Optimizing system configurations quickly by guessing at the performance. In Int’l Conference on Measurement and Modeling of Computer Systems (SIGMETRICS), 2007.

Digital Library

[49]

C. Pahl, P. Jamshidi, and O. Zimmermann. Architectural principles for cloud software. ACM Trans. on Internet Technology (TOIT), 2017.

Digital Library

[50]

C. E. Rasmussen and C. K. Williams. Gaussian processes for machine learning, volume 1. MIT press Cambridge, 2006.

Digital Library

[51]

E. B. Roecker. Prediction error and its estimation for subset-selected models. Technometrics, 33(4):459–468, 1991.

Digital Library

[52]

M. T. Rosenstein, Z. Marx, L. P. Kaelbling, and T. G. Dietterich. To transfer or not to transfer. In NIPS 2005 Workshop on Transfer Learning, volume 898, 2005.

[53]

S. Ruder and B. Plank. Learning to select data for transfer learning with Bayesian Optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.

[54]

SaC compiler. www.sac-home.org.

[55]

A. Sarkar, J. Guo, N. Siegmund, S. Apel, and K. Czarnecki. Cost-efficient sampling for performance prediction of configurable systems. In Proc. Int’l Conf. Automated Software Engineering (ASE), pages 342–352. IEEE, November 2015.

[56]

B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2016.

[57]

J. Siegmund, N. Siegmund, and S. Apel. Views on internal and external validity in empirical software engineering. In Proc. Int’l Conf. Software Engineering (ICSE), volume 1, pages 9–19. IEEE, 2015.

Digital Library

[58]

N. Siegmund, A. Grebhahn, S. Apel, and C. Kästner. Performance-influence models for highly configurable systems. In Proc. Europ. Software Engineering Conf. Foundations of Software Engineering (ESEC/FSE), pages 284–294. ACM, August 2015.

Digital Library

[59]

N. Siegmund, S. S. Kolesnikov, C. Kästner, S. Apel, D. Batory, M. Rosenmüller, and G. Saake. Predicting performance via automated feature-interaction detection. In Proc. Int’l Conf. Software Engineering (ICSE), pages 167–177. IEEE, 2012.

Digital Library

[60]

N. Siegmund, S. Sobernig, and S. Apel. Attributed variability models: Outside the comfort zone. In Proc. Int’l Symp. Foundations of Software Engineering (FSE). ACM, 2017.

Digital Library

[61]

ESEC/FSE ’18, November 4–9, 2018, Lake Buena Vista, FL, USA Pooyan Jamshidi, Miguel Velez, Christian Kästner, and Norbert Siegmund

[62]

C. Stewart, K. Shen, A. Iyengar, and J. Yin. Entomomodel: Understanding and avoiding performance anomaly manifestations. In Proc. Int’l Symp. on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 3–13. IEEE, 2010.

Digital Library

[63]

J. Styles, H. H. Hoos, and M. Müller. Automatically configuring algorithms for scaling performance. In Learning and Intelligent Optimization, pages 205–219. Springer, 2012.

[64]

E. Thereska, B. Doebel, A. X. Zheng, and P. Nobel. Practical performance models for complex, popular applications. In SIGMETRICS Perform. Eval. Rev., volume 38, pages 1–12. ACM, 2010.

Digital Library

[65]

P. Valov, J.-C. Petkovich, J. Guo, S. Fischmeister, and K. Czarnecki. Transferring performance prediction models across different hardware platforms. In Proc. Int’l Conf. on Performance Engineering (ICPE), pages 39–50. ACM, 2017.

Digital Library

[66]

D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang. Automatic database management system tuning through large-scale machine learning. In Proceedings of the International Conference on Management of Data, pages 1009–1024. ACM, 2017.

Digital Library

[67]

X. Wang, T.-K. Huang, and J. Schneider. Active transfer learning under model shift. In International Conference on Machine Learning, pages 1305–1313, 2014.

Digital Library

[68]

M. Woodside, G. Franks, and D. C. Petriu. The future of software performance engineering. In Future of Software Engineering (FOSE), pages 171–187. IEEE, 2007.

Digital Library

[69]

F. Wu, W. Weimer, M. Harman, Y. Jia, and J. Krinke. Deep parameter optimisation. In Proc. of the Annual Conference on Genetic and Evolutionary Computation, pages 1375–1382. ACM, 2015.

Digital Library

[70]

B. Xi, Z. Liu, M. Raghavachari, C. H. Xia, and L. Zhang. A smart hill-climbing algorithm for application server configuration. In 13th International Conference on World Wide Web (WWW), pages 287–296. ACM, 2004.

Digital Library

[71]

T. Xu, L. Jin, X. Fan, Y. Zhou, S. Pasupathy, and R. Talwadker. Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software. In Proc. Int’l Symp. Foundations of Software Engineering (FSE), pages 307–319, New York, NY, USA, August 2015. ACM.

Digital Library

[72]

T. Ye and S. Kalyanaraman. A recursive random search algorithm for largescale network parameter configuration. In Int’l Conference on Measurement and Modeling of Computer Systems (SIGMETRICS), pages 196–205. ACM, 2003.

Digital Library

[73]

N. Yigitbasi, T. L. Willke, G. Liao, and D. Epema. Towards machine learningbased auto-tuning of mapreduce. In Proc. Int’l Symp. on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 11–20. IEEE, 2013.

Digital Library

[74]

Y. Zhang, J. Guo, E. Blais, and K. Czarnecki. Performance prediction of configurable software systems by Fourier learning. In Proc. Int’l Conf. Automated Software Engineering (ASE), pages 365–373. IEEE, 2015.

[75]

W. Zheng, R. Bianchini, and T. D. Nguyen. Automatic configuration of internet services. ACM SIGOPS Operating Systems Review, 41(3):219–229, 2007.

Digital Library

Cited By

Chen TLi M(2024)Adapting Multi-objectivized Software Configuration TuningProceedings of the ACM on Software Engineering10.1145/36437511:FSE(539-561)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643751
Gong JChen T(2024)Predicting Configuration Performance in Multiple Environments with Sequential Meta-LearningProceedings of the ACM on Software Engineering10.1145/36437431:FSE(359-382)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643743
Acher MCombemale BRandrianaina GJezequel J(2024)Embracing Deep Variability For Reproducibility and ReplicabilityProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663621(30-35)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3641525.3663621
Show More Cited By

Index Terms

Learning to sample: exploiting similarities across environments to learn performance models for configurable systems
1. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Performance-influence models for highly configurable systems
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Almost every complex software system today is configurable. While configurability has many benefits, it challenges performance prediction, optimization, and debugging. Often, the influences of individual configuration options on performance are ...
Sampling Effect on Performance Prediction of Configurable Systems: A Case Study
ICPE '20: Proceedings of the ACM/SPEC International Conference on Performance Engineering

Numerous software systems are highly configurable and provide a myriad of configuration options that users can tune to fit their functional and performance requirements (e.g., execution time). Measuring all configurations of a system is the most obvious ...
Learning input-aware performance models of configurable systems: An empirical evaluation
Abstract
Modern software-based systems are highly configurable and come with a number of configuration options that impact the performance of the systems. However, selecting inappropriate values for these options can cause long response time, high CPU ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

October 2018

987 pages

ISBN:9781450355735

DOI:10.1145/3236024

General Chair:
Gary T. Leavens
University of Central Florida, USA
,
Program Chairs:
Alessandro Garcia
PUC-Rio, Brazil
,
Corina S. Păsăreanu
NASA Ames Research Center, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation
AFRL and DARPA
DFG

Conference

ESEC/FSE '18

Sponsor:

SIGSOFT

ESEC/FSE '18: 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 4 - 9, 2018

FL, Lake Buena Vista, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
866
Total Downloads

Downloads (Last 12 months)137
Downloads (Last 6 weeks)17

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen TLi M(2024)Adapting Multi-objectivized Software Configuration TuningProceedings of the ACM on Software Engineering10.1145/36437511:FSE(539-561)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643751
Gong JChen T(2024)Predicting Configuration Performance in Multiple Environments with Sequential Meta-LearningProceedings of the ACM on Software Engineering10.1145/36437431:FSE(359-382)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643743
Acher MCombemale BRandrianaina GJezequel J(2024)Embracing Deep Variability For Reproducibility and ReplicabilityProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663621(30-35)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3641525.3663621
Chen PChen TLi M(2024)MMO: Meta Multi-Objectivization for Software Configuration TuningIEEE Transactions on Software Engineering10.1109/TSE.2024.338891050:6(1478-1504)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3388910
Nakamori TMatsuura SMiyazaki TNakazono SSato THoshino TKawashima H(2024)ChimeraTL: Transfer Learning in DBMS with Fewer Samples2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW)10.1109/ICDEW61823.2024.00046(310-316)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDEW61823.2024.00046
Li YBao LHuang KWu CLi X(2024)RSFIN: A Rule Search-based Fuzzy Inference Network for performance prediction of configurable software systemsJournal of Systems and Software10.1016/j.jss.2023.111913209(111913)Online publication date: Mar-2024
https://doi.org/10.1016/j.jss.2023.111913
Lesoil LSpieker HGotlieb AAcher MTemple PBlouin AJézéquel J(2024)Learning input-aware performance models of configurable systems: An empirical evaluationJournal of Systems and Software10.1016/j.jss.2023.111883208(111883)Online publication date: Feb-2024
https://doi.org/10.1016/j.jss.2023.111883
Gheibi OWeyns D(2023)Dealing with Drift of Adaptation Spaces in Learning-based Self-Adaptive Systems Using Lifelong Self-AdaptationACM Transactions on Autonomous and Adaptive Systems10.1145/363642819:1(1-57)Online publication date: 13-Dec-2023
https://dl.acm.org/doi/10.1145/3636428
Iqbal MZhong ZAhmad IRay BJamshidi P(2023)CAMEOProceedings of the 2023 ACM Symposium on Cloud Computing10.1145/3620678.3624791(555-571)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3620678.3624791
Chen ZChen PWang PYu GHe ZMai GChandra SBlincoe KTonella P(2023)DiagConfig: Configuration Diagnosis of Performance Violations in Configurable Software SystemsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616300(566-578)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616300
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents