Two improving approaches for faulty interaction localization using logistic regression analysis
Pages 1039 - 1073
Abstract
Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. FROGa attempts to estimate suspiciousness more accurately using logistic regression analysis. FROGb attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that FROGa has very high accuracy, and FROGb can drastically reduce time cost for targets that have been difficult to complete by the conventional method.
References
[1]
Basili VR, Briand LC, and Melo WL A validation of object oriented metrics as quality indicators IEEE Trans. on Software Engineering 1996 22 10 751-761
[2]
Bonn, J., Foegen, K., & Lichter, H. (2019). A framework for automated combinatorial test generation, execution, and fault characterization. In: Pros. of 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 224–233.
[3]
Briand LC, Melo WL, and Wust J Assessing the applicability of fault-proneness models across object-oriented software projects IEEE Trans. on Software Engineering 2002 28 7 706-720
[4]
Cohen MB, Dwyer MB, and Shi J Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach IEEE Transactions on Software Engineering 2008 34 5 633-650
[5]
Colbourn CJ and McClary DW Locating and detecting arrays for interaction faults Journal of combinatorial optimization 2008 15 1 17-48
[6]
Colbourn CJ and Syrotiuk VR On a combinatorial framework for fault characterization Mathematics in Computer Science 2018 12 4 429-451
[7]
Cramer J The origins of logistic regression 2002 Tinbergen Institute Discussion Papers Tinbergen Institute
[8]
Fouché, S., Cohen, M. B., & Porter, A. (2009). Incremental covering array failure characterization in large configuration spaces. In: Proceeding of the 18th International Symposium on Software Testing and Analysis, ser. ISSTA ’09, pp. 177–188.
[9]
Gargantini, A., Petke, J., & Radavelli, M. (2017). Combinatorial interaction testing for automated constraint repair. In: Pros. of the 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 239–248.
[10]
Ghandehari, L. S., Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2015). Ben: A combinatorial testing-based fault localization tool. In: Proceeding of IEEE 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–4.
[11]
Ghandehari, L. S. G., Lei, Y., Xie, T., Kuhn, R., & Kacker, R. (2012). Identifying failure-inducing combinations in a combinatorial test set. In: Proceeding of IEEE 5th International Conference on Software Testing, Verification and Validation (ICST), pp. 370–379.
[12]
Ghandehari LSh, Lei Y, Kacker R, Kuhn R, Xie T, and Kung D A combinatorial testing-based approach to fault localization IEEE Transactions on Software Engineering 2020 46 6 616-645
[13]
Hagar JD, Wissink TL, Kuhn DR, and Kacker RN Introducing combinatorial testing in a large organization Computer 2015 48 4 64-72
[14]
Jayaram, R., & Krishnan, R., Approaches to fault localization in combinatorial testing: A survey. In: Smart Computing and Informatics, pp. 533–540.
[15]
Jin H, Kitamura T, Choi E, and Tsuchiya T A satisfiability-based approach to generation of constrained locating arrays Proceeding of 2018 IEEE International Conference on Software Testing 2018 ICST Workshops Verification and Validation Workshops 285-294
[16]
Kamei Y, Fukushima T, McIntosh S, Yamashita K, Ubayashi N, and Hassan AE Studying just-in-time defect prediction using cross-project models Empirical Software Engineering 2016 21 5 2072-2106
[17]
Kuhn, D., & Reilly, M. (2003). An investigation of the applicability of design of experiments to software testing. In: Proceeding of 27th Annual NASA Goddard/IEEE Software Engineering Workshop, pp. 91–95.
[18]
Kuhn DR, Kacker RN, and Lei Y Introduction to combinatorial testing 2013 CRC Press
[19]
Li, J., Nie, C., & Lei, Y. Improved delta debugging based on combinatorial testing. In: Proceeding of 12th International Conference on Quality Software, pp. 102–105.
[20]
Manning CD, Raghavan P, and Schütze H Introduction to Information Retrieval 2008 Cambridge University Press
[21]
Martínez C, Moura L, Panario D, and Stevens B Locating errors using elas, covering arrays, and adaptive testing algorithms SIAM Journal on Discrete Mathematics 2009 23 4 1776-1799
[22]
Nagamoto, T., Kojima, H., Nakagawa, H., & Tsuchiya, T. (2014). Locating a faulty interaction in pair-wise testing. In: Proceeding of IEEE 20th Pacific Rim International Symposium on Dependable Computing, pp. 155–156.
[23]
Nie C and Leung H A survey of combinatorial testing ACM Computing Surveys 2011 43 2 11
[24]
Nie C and Leung H The minimal failure-causing schema of combinatorial testing IEEE Trans. on Software Engineering 2011 20 4 1-38
[25]
Nishiura, K., Choi, E. -H., & Mizuno, O. (2017). Improving faulty interaction localization using logistic regression. In: Proceeding of 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 138–149.
[26]
Niu, X., Nie, C., Lei, Y., & Chan, A. T. (2013). Identifying failure-inducing combinations in a combinatorial test set. In: Proceeding of IEEE 6th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 271–280.
[27]
Niu X, Nie C, Leung H, Lei Y, Wang X, Xu J, and Wang Y An interleaving approach to combinatorial testing and failure-inducing interaction identification IEEE Trans. on Software Engineering 2020 46 6 584-615
[28]
Peng C -Y J, Lee KL, and Ingersoll GM An introduction to logistic regression analysis and reporting The Journal of Educational Research 2002 96 1 3-14
[29]
Pict: Pairwise independent combinatorial tool. http://github.com/Microsoft/pict
[30]
scikit-learn. http://scikit-learn.org/
[31]
Shakya, K., Xie, T., Li, N., Lei, Y., Kacker, R., & Kuhn, R. (2012). Isolating failure-inducing combinations in combinatorial testing using test augmentation and classification. In: Proceeding of IEEE 5th International Conference on Software Testing, Verification and Validation, pp. 620–623.
[32]
Shi, L., Nie, C., & Xu, B. (2005). A software debugging method based on pairwise testing. In: Proceeding of 2005 International Conference on Computational Science (ICCS), pp. 55–81.
[33]
Wang, Z., Xu, B., Chen, L., & Xu, L. (2010). Adaptive interaction fault location based on combinatorial testing. In: Proceeding of IEEE 10th International Conference on Quality Software (QSIC), pp. 495–502.
[34]
Yilmaz C, Cohen MB, and Porter AA Covering arrays for efficient fault characterization in complex configuration spaces IEEE Transactions on Software Engineering 2006 32 1 20-34
[35]
Yilmaz C, Dumlu E, Cohen MB, and Porter A Reducing masking effects in combinatorialinteraction testing: A feedback drivenadaptive approach IEEE Transactions on Software Engineering 2014 40 1 43-66
[36]
Zeller A and Hildebrandt R Simplifying and isolating failure-inducing input IEEE Transactions on Software Engineering 2002 28 2 183-200
[37]
Zhang, Z., & Zhang, J. (2011). Characterizing failure-causing parameter interactions by adaptive testing. In: Proceeding of ACM 20th International Symposium on Software Testing and Analysis (ISSTA), pp. 331–341.
[38]
Zhang, J., Ma F., & Zhang, Z. (2012). Faulty interaction identification via constraint solving and optimization. In: Proceeding of International Conference on Theory and Applications of Satisfiability Testing (SAT), pp. 186–199.
[39]
Zheng W, Wu X, Hu D, and Zhu Q Locating minimal fault interaction in combinatorial testing Advances in Software Engineering 2016 2016 10
Recommendations
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied ComputingThe behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
Generating combinatorial test suite for interaction relationship
SOQUA '07: Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meetingCombinatorial testing could detect the faults triggered by the interactions among factors in software. But in many cases, the pair-wise, N-way and even the variable strength combinatorial testing may lead test suite redundancy and fault detect ability ...
Finding Interaction Faults Adaptively Using Distance-Based Strategies
ECBS '11: Proceedings of the 2011 18th IEEE International Conference and Workshops on Engineering of Computer-Based SystemsSoftware systems are typically large and exhaustive testing of all possible input parameters is usually not feasible. Testers select tests that they anticipate may catch faults, yet many unanticipated faults may be overlooked. This work complements ...
Comments
Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 07 June 2024
Accepted: 06 February 2024
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 26 Jan 2025