See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/342972286

Artificial Intelligence Applied to Software Testing: A Literature Review

Conference Paper · June 2020

DOI: 10.23919/CISTI49556.2020.9141124

CITATIONS READS

5 2,960

3 authors, including:

Rui Lima António Miguel Rosado da Cruz

Instituto Politécnico de Viana do Castelo Instituto Politécnico de Viana do Castelo
7 PUBLICATIONS   48 CITATIONS    44 PUBLICATIONS   416 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Automatic Generation of User Interfaces from Rigorous Domain and Use Case Models - PhD research project View project

Amália Project View project

All content following this page was uploaded by Rui Lima on 26 July 2020.

The user has requested enhancement of the downloaded file.

Artificial Intelligence Applied to Software Testing: A
Literature Review
Rui Lima, António Miguel Rosado da Cruz, Jorge Ribeiro
Instituto Politécnico de Viana do Castelo
Rua Escola Industrial e Comercial de Nun’Álvares, 34
4900-347, Viana do Castelo, Portugal
ruirochalima1@gmail.com, miguel.cruz@estg.ipvc.pt, jribeiro@estg.ipvc.pt

Abstract — In the last few years Artificial Intelligence (AI) testing, however the lack of an oracle (the mechanism that
algorithms and Machine Learning (ML) approaches have been distinguishes correct and incorrect behavior of the SUT) is
successfully applied in real-world scenarios like commerce, currently a bottleneck [6]. AI is still rarely used for error
industry and digital services, but they are not a widespread detection in SUT because of the lack of automation of the
reality in Software Testing. Due to the complexity of software oracle problem. The exception is regression testing where the
testing, most of the work of AI/ML applied to it is still academic. correct performance of the SUT can be derived from its
This paper briefly presents the state of the art in the field of previous version behavior [7].
software testing, applying ML approaches and AI algorithms.
The progress analysis of the AI and ML methods used for this In this context, this paper briefly presents the state of the art
purpose during the last three years is based on the Scopus and explores the available literature in software testing
Elsevier, web of Science and Google Scholar databases. applying AI/ML approaches/algorithms. The progress of the AI
Algorithms used in software testing have been grouped by test and ML methods used for this purpose during the last three
types. The paper also tries to create relations between the main years is analyzed. The paper also tries to create relations
AI approaches and which type of tests they are applied to, in between the main AI approaches and which type of tests they
particular white-box, grey-box and black-box software testing are applied to. For this study, research articles were
types. We conclude that black-box testing is, by far, the preferred downloaded from databases such as Scopus, Elsevier, web of
method of software testing, when AI is applied, and all three
Science and Google Scholar, between the year 2017 and
methods of ML (supervised, unsupervised and reinforcement) are
December 2019. The search terms used have been “artificial-
commonly used in black-box testing being the “clustering”
technique, Artificial Neural Networks and Genetic Algorithms intelligence”, “software-testing” and both these keywords. It
applied to “fuzzing” and regression testing. was not used the term “Machine Learning” by the fact that
most of the papers found implemented AI techniques (or
Keywords – Software Testing, Test Pattern, Artificial hybrid [8]) and not specifically using a ML approach (as
Intelligence, Machine Learning, Artificial Neural Network, supervised, unsupervised or reinforced learning), where one of
Genetic Algorithm. these learning types can be implemented with one or more AI
algorithms.
I. INTRODUCTION
The paper is organized as follows. In Section II we present
Software testing is the main way of validating software an overview of the ML approaches, AI algorithms (and a brief
against the defined requirements and accounts for about half of relation between the ML and AI) and software test types
the cost and time of development [1] and it is argued that referred in this paper. Section III presents an overview of AI
improvements in system infrastructure could save up to a third applied to software testing. The results are analyzed and
of its costs [2]. On the other hand, in the last decades the discussed in Section IV. In Section V, a conclusion is given.
concepts of Artificial Intelligence (AI) and Machine Learning
(ML) have been successfully used to explore the potentialities II. DEFINITIONS
of the data in different fields [3]. ML [4] is used to teach In this section, the most popular Artificial Intelligence (AI)
machines how to handle the data more efficiently simulating algorithms and Machine Learning (ML) approaches are
the learning concept of the rational beings and can be imple- described with the main focus on the software testing field.
mented with AI algorithms (or techniques) reflecting the para-
digms/approaches of the rational characteristics as connection- From the origins, the definitions of AI and ML are strongly
nist, genetic, statistical and probabilities, case based, etc. With connected. John McCarthy (1959), defined AI algorithms as “a
the AI algorithms, and based on ML approach, it is possible to branch of computer science dealing with the simulation of
explore and extract information in order to classify, associate, intelligent behavior in computers” [3], and Tom Mitchell
optimize, clustering, forecasting, identify patterns, etc. (1997) defined Machine Learning as “the study of computer
algorithms that allow computer programs to automatically
It is known that AI techniques provide predictive models to improve through experience” [4]. Following the learning
be used for multiple engineering purposes [5], but it still isn’t a concept of rational beings, ML is structured in learning types
popular implementation for checking Systems Under Test (supervised, unsupervised and reinforcement learning) and AI
(SUT) correctness. It seems logic to apply AI into software is organized in paradigms/approaches of the rational
characteristics as connectionist, genetic, statistical and information. A simple ANN only has one hidden layer while a
probabilities, case based or inductive rules materialized with deep ANN has two or more (Figure 1).
Artificial Neural Networks, Genetic Algorithms, naive Bayes,
Case Based Reasoning, inductive rules, or decision trees. With The unit of ANN is the neuron and the connections
the AI algorithms and based on a ML approach it is possible to between neurons are called links or synapses. The neuron takes
explore and extract information to classify, associate, optimize, a number of inputs 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∈ ℝ and give an output 𝑜 ∈ ℝ.
clustering, forecasting, identify patterns, etc. The most common design for a neuron is the perceptron and
can be represented by 2 equations: the transfer function and the
A. Machine Learning Approaches activation function [10]. The transfer function is as:
Machine Learning approaches (or methods) are classified in
three main categories and, for each ML approach, there’s one z(xi )=b+ ∑ni=1 wi xi ()
or more AI algorithm that implements the learning approach:
Where b is the bias constant, which can be used to delay the
1) Supervised Learning: In order to make predictions of a triggering of the activation function (that might help to create a
given set of samples, the training data is given all the right better model), and 𝑤𝑖 represents the weight given to the input.
answers for the target variable, and the algorithm’s job is to
replicate the right answer based on the data set [13]. This type
of learning is used for classification and regression purposes
of AI objetives. Thus, the objective of this method is to
classify the free parameters while the output errors are
minimized [17]. So, the set might be separated into training
data and testing data, to calculate accuracy and loss of a
particular algorithm.
2) Unsupervised Learning: This method is used to discover
hidden patterns in data that has no target variable (expected
outcomes) and organizes the data into a group of clusters to
describe its structure. In unsupervised learning no training data
is provided to the system, and is usually more challenging than
the other methods, so it is more often used for data exploration
[3] [4] [13] [15]. Algorithms commonly used for this method
might be k-means (where the user specifies the desired
number of clusters) or hierarchical clustering (where the user Figure 1. Diagram of a Deep ANN
doesn’t need to commit to a particular number of clusters),
both of which attempt to group objects together based on their There are multiple types of activation functions. One of the
most commons is the binary step activation function:
similarity [14]. This type of learning is used for density
extraction of the data and clustering purposes of AI objetives. 1, z>0
3) Reinforcement Learning: Here, the algorithm is constan- o(z)= { ()
-1, z≤0
tly adapting to the environment, based on its feedback, which
might be positive or negative [21]. The algorithm chooses an In this case the activation function outputs a 1 or -1,
action based on each item of the data set and later learns the depending on a threshold (in this case 0) [10].
efectiveness of the decision, changing its strategy to learn
The weight (for each perceptron) starts as a random guess
better and achieve the best reward for the action taken [4].
and is updated according to the following rule:
B. Artificial Intelligence Algorithms
Besides the large number of AI algorithms (or techniques), Δwi =η(t-o)xi ()
when applied to software testing the studies mainly included
Artificial Neural Networks (including Support Vector Where 𝜂 is the learning rate (a positive constant), t is the
Machines - SVM), k-Nearest Neighbour (kNN) approach, target output, o is the current output for the input 𝑥𝑖 . If t and o
naive Bayes classifier, decision trees and rules induction are equal the desired output is reached, and the weight is not
algorithm. In this section we focus on these AI algorithms: updated anymore [10].
1) Artificial Neural Network: Artificial Neural Networks 2) Genetic Algorithm: A Genetic Algorithm (GA) is an
(ANN) are function approximators commonly used to optimization strategy that mimics the principles of Darwinian
approximate continuous states and actions. Typically, they are theory of evolution. GA is commonly used to solve non-linear,
better suited to classify large space states that couldn’t be multimodal and discontinuous optimization problems [11].
processed otherwise [9]. They are composed of an input layer The algorithm needs to have two things defined: a fitness
(with the input data), an output layer (capable of classifying the function, capable of evaluating the best solutions for a
data) and one or more hidden layers that process the
problem, and a way to represent the “DNA” of the candidate the help of ANN [14]. Autoencoders can be used in a variety of
solutions. problems, such as: anomaly detection, denoising, pretraining,
similarity detection or manual output generation. Autoencoders
are usually an unsupervised means to a supervised end [14].
Another AI algorithm/technique is Bayesian networks, which is
also an unsupervised algorithm [16], but unlike machine
learning solutions (that can only deduce correlation in
datasets), it can be used to try to infer causation between
features.
With Reinforcement Learning, unlike normal machine
learning, the objective is to reach a goal and not to look at
training datasets. The environment itself might be known or
unknown depending on the used algorithm (model-based or
model-free algorithms, respectively). Examples of commonly
used algorithms are Q-Learning and Deep Q-Learning
Networks (DQN) [21].
C. Software Testing Methods
A brief definition of the software testing methods, from the
point of view of code visibility and analysis (white-box, grey-
box and black-box), is given in this section:
1) White-Box Testing: The detailed analysis of a program’s
code, such as internal logic or structure. In white-box testing,
Figure 2. GA Flowchart.
the tester, needs to have full access to the source code to find
which part of it is having an unwanted behaviour. This method
The workflow of a GA (refer to Figure 2) starts with the has a high granularity, so it is potentially the most time-
initialization of the candidate solutions, and then the evaluation consuming method and also requires skilled testers. On the
of those solutions by the fitness function. If the termination other hand, it allows for maximum coverage during test
criteria are met, the optimal solution is presented. Otherwise scenario writing. It is applicable at unit, integration and system
the most promising candidates are selected to be modified. The levels of the software testing process, and common techniques
candidates change involves crossover and then mutation. are basis path and control flow testing [22].
Crossover generates new candidates by combining existing 2) Black-Box Testing: This approach tests the application
ordered segments from existing solutions, while mutation
randomly changes parts of an existing solution into a new one. based on its results, at several levels, without having any
This process is repeated until the optimal solution is meet [11]. knowledge of its internal working. Although this method has
limited coverage it is the least time consuming and the test
When solving a GA problem, instead of coding a specific perception is very simple. Commonly used techniques are
solution, one should provide rules that a solution must fulfill in fuzzing (injecting malformed data for finding implementation
order to be accepted (allowing to solve problems with a large
bugs) or state transitioning testing (testing state machines and
quantity of potential solutions) [12].
graphical user interface navigation) [22].
For each ML approach there’s one or more AI algorithms 3) Grey-Box Testing: This method is used to test an
that implements the learning approach. As we mentioned application with limited knowledge of its internal working and
before, Supervised Learning is used to solve classification full knowledge of the fundamental aspects of the system. In
problems (for target variables with discrete values) or grey-testing the modules are studied for the design of test
regression problems (for target variables with continuous
cases (white-box), but the actual tests are performed in the
values) [18]. Examples of AI algorithms used with this
approach are naive bayes, decision trees, support vector machi- exposed interfaces (black-box). Since source code is not
nes (SVM), ANN, recurrent neural networks (RNN) or convo- available the coverage is limited. Some forms of grey-box
lutional neural networks (CNN). RNNs are usually applied in testing are regression testing (checking if new changes are
sequential data [19] and CNNs are common solutions for large made to the software) and pattern testing (verifying the
scale inputs [20], such as sequential data or images. architecture and design of the application) [22].
For Unsupervised Learning, the algorithms commonly used III. AI APPLIED TO SOFTWARE TESTING – SURVEY RESULTS
are the k-means (where the user specifies the desired number of
clusters) or hierarchical clustering (where the user doesn’t need This section is divided into three parts. Subsection A
to commit to a particular number of clusters), both of which presents an overview of the research published about AI and
attempt to group objects together based on their similarity [14]. software testing. Subsection B shows a more detailed analysis
Autoencoders can also be used for feature extraction, of research works referencing AI algorithms and software
generating new features from the original set of features with testing methods. Subsection C tries to scrutinize the most
popular test cases involving AI and describe some examples and Web of Science (https://apps.webofknowledge.com). The
observed in some of the papers. searches have been done for documents with the keywords
“artificial-intelligence” (Figure 3), “software-testing” (Figure
4) and with both the keywords (Figure 5), up to the year 2019.
B. Detailed View
For a more detailed analysis, the total number of papers is
determined for combinations between AI learning methods and
software testing methods (omitted combinations mean that no
papers were found). The used AI learning methods are
supervised, unsupervised and reinforcement learning. The used
software testing methods are white, black and grey-box. The
number of papers is the total sum in the last 3 years, from 2017
up until the time of this writing (January 2020). The measure is
made for each of the three search engines: Google Scholar
(Figure 6), Scopus (Figure 7) and Web of Science (Figure 8).

Figure 3. AI Research Published by Year and Source

Figure 6. Number of Articles with the Keyword Combination of AI and

Software Testing Methods from 2017 to January 2020 for Google Scholar

Figure 4. Software Testing Research Published by Year and Source

Figure 7. Number of Articles with the Keyword Combination of AI and

Software Testing Methods from 2017 to January 2020 for Scopus Elsevier

Figure 8. Number of Articles with the Keyword Combination of AI and

Software Testing Methods from 2017 to January 2020 for Web of Science

C. Test Patterns
Figure 5. Research with the Keywords Software-Testing and Artificial-
Intelligence Published by Year and Source Specific AI algorithms used in software testing are explored
to check their popularity. The AI algorithms in question are:
A. Overview clustering, encoders, decision trees, ANN, support vector
In order to have a broad view of the AI and software testing machines, naïve bayes, bayesian networks, DQN and GA.
research done over time, some scientific search engines and Keywords for those algorithms are used in conjunction with the
repositories were analysed. Used repositories and papers’ “software testing” keyword to have an idea of the most favored
search engines have been Google Scholar (omitted combinations mean that no papers were found). The
(https://scholar.google.com), Scopus (https://www.scopus.com) number of articles is the sum from 2017 to January 2020 for
Google Scholar (Figure 9), Scopus (Figure 10) and Web of
Science (Figure 11).

Figure 12. Number of Articles with Software Testing Algorithm Keywords

used in AI from 2017 to January 2020 for Google Scholar

Figure 9. Number of Articles with AI Algorithm Keywords used in Software

Testing from 2017 to January 2020 for Google Scholar Figure 13. Number of Articles with Software Testing Algorithm Keywords
used in AI from 2017 to January 2020 for Scopus Elsevier

Figure 14. Number of Articles with Software Testing Algorithm Keywords

used in AI from 2017 to January 2020 for Web of Science

Some popular papers [23][24][25] present methods that

make the code exploration more efficient. The tools extract
automatically the code interface and generate path tests
following a determined algorithm. One of the papers [24] uses
a tool (scalable automated guided execution or SAGE) that
Figure 10. Number of Articles with AI Algorithm Keywords used in Software implements a search algorithm, called “generational search”,
Testing from 2017 to January 2020 for Scopus Elsevier
that tries to cover the maximum output using the minimum
input combinations, to save the number of necessary tests. In
another case [23] the tool (directed automated random testing
or DART) is capable of extracting automatically the program
interface, generate path tests and make a dynamic analysis to
verify the code behavior and generate new tests and input data.
A Microsoft program (Program Exploration tool or Pex) for
exploratory white-box unit testing is also referred in a paper
[25] and is capable of automatically generating new paths.
An example of AI in regression testing is the use of ANN in
reinforcement learning, presented in [9], allowing the algorithm
to adapt when new test cases are added or removed. This
pattern is capable of selecting and giving priority to test cases
according to their duration, last execution and error history. In
Figure 11. Number of Articles with AI Algorithm Keywords used in Software
[26], an algorithm is proposed (MultiWalk) for being used in
Testing from 2017 to January 2020 for Web of Science the reduction of the set of tests to reduce their execution time
as a whole. Another paper [27] investigates a group of
Software testing techniques used in AI are also explored in optimization algorithms for test suit reduction. Those
the same manner referenced in the previous paragraph. The algorithms are compared between themselves and a random
examined techniques are: basis path testing, data flow testing, search algorithm (used for reference).
fuzzing, regression testing and pattern testing. The time
interval is also from 2017 to January 2020 and the researched IV. RESULT ANALYSIS
sources are the same: Google Scholar (Figure 12), Scopus Figures 3, 4 and 5 show that the keywords “artificial
(Figure 13) and Web of Science (Figure 14). If no papers were intelligence” and “software testing” gained an exponential
found for a particular combination, the bar is omitted. popularity in the last few decades. But, the last year (2019)
presents a drop of research made about those subjects. Papers
with both keywords present follow the same pattern. Figure 6, [3] Haenlein, Michael & Kaplan, Andreas. (2019). A Brief History of
7 and 8 show that, in the last three years, the AI solutions Artificial Intelligence: On the Past, Present, and Future of Artificial
Intelligence. California Management Review.
where mostly applied to “black-box” testing methods. For
[4] Mitchell, T. M. (1997). Machine Learning. New York: McGraw-Hill.
Google Scholar, the most popular ML method (in the past 3 ISBN: 978-0-07-042807-2.
years) used in software testing is supervised learning, followed [5] Groz, R., Simao, A., Bremond, N., & Oriat, C. (2018). Revisiting AI and
by reinforcement learning, however, for Scopus Elsevier and testing methods to infer FSM models of black-box systems. In
Web of Science reinforcement learning was the most popular. IEEE/ACM 13th Int. Work. on Automation of Software Test,(pp.16-19).
Although not as popular as the two methods referenced above, [6] Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2014).
unsupervised learning shows to be a very common occurrence The oracle problem in software testing: A survey. IEEE transactions on
in black-box testing. White-box and grey-box testing do not software engineering, 41(5), (pp:507-525).
seem to use AI quite as often. [7] Shahamiri, S. R., Kadir, W. M. N. W., & Mohd-Hashim, S. Z. (2009). A
comparative study on automated software test oracle methods. In 2009
The most popular AI algorithms appear to be “clustering” 4th Int. Conf. on Software Engineering Advances (pp. 140-145).
(a common unsupervised learning method), ANN (used in [8] Shafiabady, N., Lee, L. H., Rajkumar, et al. (2016). Using unsupervised
supervised and reinforcement learning) and GA (commonly clustering approach to train the support vector machine for text
classification. Neurocomputing, 211, (pp:4–10).
used in reinforcement learning), as shown in Figures 9, 10 and
11. The most common software testing types performed by AI [9] Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2018).
Reinforcement learning for automatic test case prioritization and
algorithms seem to be “fuzzing” and regression testing (Figures selection in continuous integration. arXiv preprint arXiv:1811.04122.
12, 13 and 14). The most cited papers read describe test [10] Kraus, D. (2018). Machine Learning and Evolutionary Computing for
patterns that correlate with the analyzed data. Some examples GUI-based Regression Testing. arXiv preprint arXiv:1802.03768.
are ANN algorithms, used with reinforcement learning, to [11] Windisch, A., Wappler, S., & Wegener, J. (2007). Applying particle
perform regression testing [9], optimization algorithms used for swarm optimization to software testing. In Procs of the 9th annual
test suit reduction [26][27] and “fuzzing” testing used for path conference on Genetic and evolutionary computation (pp. 1121-1128).
exploration in code [23][24][25]. [12] Sheppard, C. (2017). Genetic algorithms with python. Smashwords Ed.
[13] Dangeti, P. (2017). Statistics for machine learning. Packt Publishing Ltd.
V. CONCLUSIONS [14] Patel, A. A. (2018). Hands-On Unsupervised Learning Using Python:
The apparent decrease of research (regarding AI and How to Build Applied Machine Learning Solutions from Unlabeled
Data. O'Reilly Media, Incorporated.
software testing) published in 2019, shown in Figures 3, 4 and
[15] Li, X., & Wong, K. C. (Eds.). (2019). Natural Computing for
5, could be due to the fact that some papers from the last year Unsupervised Learning. Springer International Publishing.
are still unindexed, or it could indicate the beginning of a third
[16] Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network
AI winter, where new AI development stalls [28]. classifiers. Machine learning, 29(2-3), pp:131-163.
Black-box testing is, by far, the software testing method [17] Suresh, S., Sundararajan, N., & Savitha, R. (2013). Supervised learning
that most benefit takes from AI. All three methods ML with complex-valued neural networks (pp. 125-132). Berlin: Springer.
(supervised, unsupervised and reinforcement) are commonly [18] Huang, T. M., Kecman, V., & Kopriva, I. (2006). Kernel based
algorithms for mining huge data sets (Vol. 1). Heidelberg: Springer.
used in black-box testing. Arguably, the most common AI
[19] Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
algorithms used for software testing are “clustering”, ANN and (2010). Recurrent neural network based language model. In 11th annual
GA, applied to “fuzzing” and regression testing. The results conf. of the international speech communication association.
show that the majority of the observed AI solutions are used as [20] Gulli, A., & Pal, S. (2017). Deep learning with Keras. Packt Publ. Ltd.
a form of optimization. The algorithms are used, in most cases, [21] Nandy, A., & Biswas, M. (2017). Reinforcement Learning: With Open
to help cover most of the code with the least amount of test AI, TensorFlow and Keras Using Python. Apress.
cases. For that to occur AI is used: (i) to find efficient ways to [22] Khan, M. E., & Khan, F. (2012). A comparative study of white box,
cover the code; (ii) prioritize more pertinent test cases; (iii) black box and grey box testing techniques. Int. J. Adv. Comput. Sci.
reduce the quantity of test cases, filtering tests with more Appl, 3(6).
coverage, less time-consuming or with a bigger fail track [23] Godefroid, P., Klarlund, N., & Sen, K. (2005). DART: directed
record. automated random testing. In Proc. of 2005 ACM SIGPLAN conference
on Programming language design and implementation (pp. 213-223).
Since the test phase is usually a very time intensive phase [24] Godefroid, P., Levin, M. Y., & Molnar, D. (2012). SAGE: whitebox
of any software project, not all tests can be fulfilled before the fuzzing for security testing. Queue, 10(1), 20-27.
publishing deadline. Because of that, it is important to apply [25] Tillmann, N., De Halleux, J., & Xie, T. (2014). Transferring an
tests in a clever manner. One of the solutions for an efficient automated test generation tool to practice: From Pex to Fakes and Code
Digger. In Procs. of the 29th ACM/IEEE Int. Conf on Automated
testing phase can be the routine application of AI in software software engineering (pp. 385-396).
testing. [26] Chi, Z., Xuan, J., et al. (2017). Multi-level random walk for software test
suite reduction. IEEE Comp. Intelligence Magazine, 12(2), (pp.24-33).
REFERENCES
[27] Sahin, O., & Akay, B. (2016). Comparisons of metaheuristic algorithms
[1] Myers, G. J., Sandler, C., & Badgett, T. (2011). The art of software and fitness functions on software test data generation. Applied Soft
testing. John Wiley & Sons. Computing, 49, 1202-1214.
[2] Planning, S. (2002). The economic impacts of inadequate infrastructure [28] Yasnitsky L.N. (2020). Whether Be New “Winter” of Artificial
for software testing. National Institute of Standards and Technology. Intelligence?. In: Antipova T. (eds) Integrated Science in Digital Age.
ICIS 2019. Lecture Notes in Networks and Systems, vol 78. Springer,
Cham.

View publication stats