research-article

Testing configurable software systems: the failure observation challenge

Authors:

Fischer Ferreira,

Markos Viggiato,

Maurício Souza,

Eduardo FigueiredoAuthors Info & Claims

SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

Article No.: 28, Pages 1 - 6

https://doi.org/10.1145/3382025.3414725

Published: 19 October 2020 Publication History

Abstract

Configurable software systems can be adapted or configured according to a set of features to increase reuse and productivity. The testing process is essential because configurations that fail may potentially hurt user experience and degrade the reputation of a project. However, testing configurable systems is very challenging due to the number of configurations to run with each test, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing techniques and tools have been proposed to deal with this challenge, but their potential practical application remains mostly unexplored. To encourage the research area on testing configurable systems, researchers and practitioners should be able to try out their solutions in common datasets. In this paper, we propose a dataset with 22 configurable software systems and an extensive test suite. Moreover, we report failures found in these systems and source code metrics to allow evaluating candidate solutions. We hope to engage the community and stimulate new and existing approaches to the problem of testing configurable systems.

References

[1]

M. Al-Hajjaji, S. Krieter, T. Thüm, M. Lochau, and G. Saake. 2016. IncLing: Efficient Product-line Testing Using Incremental Pairwise Sampling. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (Amsterdam, Netherlands) (GPCE 2016). ACM, New York, NY, USA, 144--155.

Digital Library

[2]

M. Al-Hajjaji, J. Meinicke, S. Krieter, R. Schröter, and T. Leich T. Saake G. Thüm. 2017. Tool demo: testing configurable systems with FeatureIDE. ACM SIGPLAN Notices 52, 3 (2017), 173--177.

Digital Library

[3]

M. Krieter S. Thüm T. Lochau M. Saake G Al-Hajjaji. 2016. IncLing: efficient product-line testing using incremental pairwise sampling. In 15th Proceedings of the ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE). 144--155.

Digital Library

[4]

S. Apel, S. Batory, C. Kästner, and G. Saake. 2013. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer.

[5]

S. R. Chidamber, D. P. Darcy, and C. F. Kemerer. 1998. Managerial use of metrics for object-oriented software: An exploratory analysis. Transactions on software Engineering (TSE) (1998), 629--639.

[6]

CK. [n.d.]. CK. https://github.com/mauricioaniche/ck, Accessed 16-nov-2019.

[7]

Myra B. Cohen, Peter B. Gibbons, Warwick B. Mugridge, and Charles J. Colbourn. 2003. Constructing test suites for interaction testing. In Proceedings of the International Conference on Software Engineering (ICSE). IEEE, 38--48.

[8]

H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque. 2016. Pit: a practical mutation testing tool for java. In In 25th International Symposium on Software Testing and Analysis (ISSTA). 449--452.

[9]

P. A. D. M. S. Neto, I. do Carmo Machado, J. D. McGregor, E. S. de Almeida, and S. R. de Lemos Meira. 2011. A Systematic Mapping Study of Software Product Lines Testing. Information and Software Technology (IST) (2011), 407--423.

[10]

E. Engström and P. Runeson. 2011. Software Product Line Testing - A Systematic Mapping Study. Information and Software Technology (IST) (2011), 2--13.

[11]

F. Ferreira, J. P. Diniz, C. Silva, and E. Figueiredo. 2019. Testing Tools for Configurable Software Systems: A Review-based Empirical Study. In Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems(VaMoS).

[12]

F. Ferreira, G. Vale, E. Figueiredo, and J. P. Diniz. 2020. On the Proposal and Evaluation of a Test-enriched Dataset for Configurable Systems. In Proceedings of the 4th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS).

[13]

Fest. [n.d.]. Fest. http://code.google.com/p/fest/, Accessed 16-nov-2019.

[14]

M. Greiler, A. Deursen, and M. Storey. 2012. Test Confessions: A Study of Testing Practices for Plug-in Systems. In Proceedings of the International Conference on Software Engineering. IEEE, 244--254.

[15]

A. Halin, A. Nuttinck, M. Acher, X. Devroey, G. Perrouin, and B. Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empirical Software Engineering (ESE) (2019), 674--717.

[16]

IncLing. [n.d.]. IncLing Tool. http://www.tinyurl.com/IncrementalSampling, Accessed 15-nov-2019.

[17]

JaCoCo. [n.d.]. Metrics. https://www.eclemma.org/jacoco/, Accessed 16-nov-2019.

[18]

M. F. Haugen Ø. Fleurey F. Johansen. 2012. An Algorithm for Generating T-wise Covering Arrays from Large Feature Models. In 16th Proceedings of the International Software Product Line Conference - Volume 1 (SPLC). 46--55.

Digital Library

[19]

JUnit. [n.d.]. JUnit. https://junit.org/junit5/, Accessed 16-nov-2019.

[20]

P. Kim, S. Khurshid, and D. Batory. 2012. Shared Execution for Efficiently Testing Product Lines. In Proceedings of the International Symposium on Software Reliability Engineering. IEEE, 221--230.

[21]

P. Kim, D. Marinov, S.z Khurshid, D. Batory, S. Souto, P. Barros, and M. d'Amorim. 2013. SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems. In 9th Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE). 257--267.

Digital Library

[22]

D. R. Kuhn, R. N. Kacker, and Y. Lei. 2010. Practical Combinatorial Testing. Technical Report SP 800-142. National Institute of Standards & Technology, Gaithersburg, MD, United States.

[23]

I. Machado, J. McGregor, Y. Cavalcanti, and E. Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology (IST) (2014), 1183 -- 1199.

[24]

I. C. Machado, J. D. Mcgregor, Y. C. Cavalcanti, and E. S. de Almeida. 2014. On Strategies for Testing Software Product Lines: A Systematic Literature Review. Information and Software Technology (IST) 56, 10 (2014), 1183--1199.

Digital Library

[25]

J. Martinez, W. KG Assunção, and T. Ziadi. 2017. ESPLA: A catalog of Extractive SPL Adoption case studies. In In 21st International Systems and Software Product Line Conference (SPLC). 38--41.

[26]

F. Medeiros, C. Kästner, M. Ribeiro, R. Gheyi, and S. Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proceedings of the International Conference on Software Engineering. ACM, 643--654.

[27]

J. Meinicke, C. Wong, C. Kästner, T. Thüm, and Gunter S. 2016. On essential configuration complexity: measuring interactions in highly-configurable systems. In 31st Proceedings of the International Conference on Automated Software Engineering (ASE). 483--494.

[28]

Metrics. [n.d.]. Metrics. http://metrics.sourceforge.net/, Accessed 16-nov-2019.

[29]

Mockito. [n.d.]. Mockito. https://site.mockito.org/, Accessed 16-nov-2019.

[30]

H. Nguyen, Christian K., and N. Nguyen. 2014. Exploring Variability-aware Execution for Testing Plugin-based Web Applications. In Proceedings of the International Conference on Software Engineering (ICSE). ACM, 907--918.

[31]

C. Nie and H. Leung. 2011. A survey of combinatorial testing. ACM Computing Surveys 43, 2 (2011), 11:1--11:29.

[32]

K. Pohl and A. Metzger. 2006. Software Product Line Testing. Information and Software Technology (IST) (2006), 78--81.

[33]

A. Santos, P. Alves, E. Figueiredo, and F. Ferrari. 2016. Avoiding code pitfalls in aspect-oriented programming. Science of Computer Programming (SCP) 119 (2016), 31--50.

Digital Library

[34]

S. Souto, M. d'Amorim, and R. Gheyi. 2017. Balancing soundness and efficiency for practical testing of configurable systems. In In 39th Proceedings of the International Conference on Software Engineering (ICSE). 632--642.

[35]

spl2go. [n.d.]. SPL2go. http://spl2go.cs.ovgu.de/projects/, Accessed 10-nov-2019.

[36]

M. Svahnberg, J. Van Gurp, and J. Bosch. 2005. A taxonomy of variability realization techniques. Software: Practice and experience (SPE) (2005), 705--754.

[37]

T. Thüm, C. Kästner, F. Benduhn, J. Meinicke, G. Saake, and T. Leich. 2014. FeatureIDE: An extensible framework for feature-oriented software development. Science of Computer Programming (SCP) (2014), 70--85.

[38]

G. Vale, D. Albuquerque, E. Figueiredo, and A. Garcia. 2015. Defining metric thresholds for software product lines: a comparative study. In Proceedings of the 19th International Conference on Software Product Line (SPLC). 176--185.

[39]

C. Wong, J. Meinicke, L. Lazarek, and C. Kästner. 2018. Faster variational execution with transparent bytecode transformation. Proceedings of the ACM on Programming Languages (OOPSLA) (2018).

Cited By

Casquina JMontecchi LMousavi MSchobbens P(2021)A proposal for organizing source code variability in the git version control systemProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume A10.1145/3461001.3471141(82-88)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461001.3471141

Testing configurable software systems: the failure observation challenge
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Testing Tools for Configurable Software Systems: A Review-based Empirical Study
VaMoS '19: Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems

Configurable software systems are software systems that can be adapted or configured according to a set of features with the goal of increasing reuse and productivity. However, testing configurable systems is very challenging due to the number of ...
On the proposal and evaluation of a test-enriched dataset for configurable systems
VaMoS '20: Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems

Configurable systems offer advantages compared to single systems since developers should maintain a unique platform to address a diversity of deployment contexts and usages. To ensure that all configurations correctly execute, developers spend ...
A firewall model for testing user-configurable software systems

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

October 2020

323 pages

ISBN:9781450375696

DOI:10.1145/3382025

General Chair:
Roberto Erick Lopez-Herrejon
École de technologie supérieure, Canada

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

CAPES
CNPq
FAPEMIG

Conference

SPLC '20

Sponsor:

SIGSOFT

SPLC '20: 24th ACM International Systems and Software Product Line Conference

October 19 - 23, 2020

Quebec, Montreal, Canada

Acceptance Rates

SPLC '20 Paper Acceptance Rate 17 of 49 submissions, 35%;

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
110
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Casquina JMontecchi LMousavi MSchobbens P(2021)A proposal for organizing source code variability in the git version control systemProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume A10.1145/3461001.3471141(82-88)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461001.3471141

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents