CODE-BASED TESTING OF EXTENDED

FINITE STATE MACHINE MODELS

Presented by
Mahitab Hassan - g00042869
Ragini Gupta - g00049089

Supervised by
Dr. Khaled El-Fakih
OUTLINE
 Introduction
 Preliminaries
 EFSM Model
 Mutants
 Test Cases and Test Suites
 Previous Work
 Fault Coverage
 Considered EFSM Examples
 Considered Faults (Mutants)
 Considered Test Cases and Test Suites
 Assessment Methodology
 Results and Ranking 2
 Conclusion
INTRODUCTION
An EFSM (Extended Finite State Machine) model M is an
extension of an FSM with the addition of variables,
predicates or guards, update statements and parameters.

M = (Q, Σ1, Σ2, I, V, T)

In the given example:

 Inputs (X): a, b

 Output (Y): o(w)

 Initial State (I) : S0

3
 Context Variable (V): w

 Transitions (T): T1, T2, T4, T5

PROBLEM STATEMENT

 Why is EFSM important?

 Provides a powerful model for deriving functional tests for
software systems, digital hardware, protocols.

 Need for test suites to cover EFSM Faults

 EFSM Based Testing approaches:

 Code-based testing,
 Fault Diagnosis,
 Test Derivation
4
OBJECTIVES
 Code Based Testing of EFSM Models

 Code Based Testing involves viewing the

software as a white box and obtaining the test
cases from the implementation of the
software/program

 Assess the fault coverage of code-based EFSM test suites

and rank them depending on coverage

5
FAULTS (MUTANTS)
 Mutants are a way of modeling faults

 M’ is a mutant of M if they are distinguishable given a

test case with input sequence 

 In code-based testing
 Added by inserting, replacing, or deleting operators
 Can be arithmetic, conditional, logical, etc.

6
CODE-BASED MUTANTS : ARITHMETIC MUTANTS
 Arithmetic Operator Replacement – Binary (AORB)
𝐴 =𝐴+5 → 𝐴 =𝐴−5

 Arithmetic Operator Replacement – Shortcut (AORS)

𝐴++ → 𝐴−−

 Arithmetic Operator Insertion – Shortcut (AOIS)

→ − −𝐴

 Arithmetic Operator Insertion – Unary (AOIU)

→ 𝐴 = −1 7
CODE-BASED MUTANTS (CONT.)
 Conditional Operator Replacement (COR)
A || 𝐵 → 𝐴 && 𝐵

 Conditional Operator Insertion (COI)

→ A || 𝐵

 Conditional Operator Deletion (COD)

A || 𝐵

8
CODE-BASED MUTANTS (CONT.)
 Relational Operator Replacement (ROR)
A>𝐵→ 𝐴<𝐵

 Logical Operator Insertion (LOI)

→A|𝐵

9
TEST CASES AND TEST SUITES
 A test case
Finite sequence of inputs and outputs derived from the
specification (or its flow-graph)

 Executable test case derived from:

Input/output pairs of a test path

 Fulfill some coverage criteria such All Uses path, DU paths,

etc.

10
EFSM FLOW-GRAPH BASED TEST SUITES
 All-Uses Test Suite
 Covers the All-Definition Use Pairs of all context variables
and parameterized input

 Used DataFlow Testing Tool for deriving All Uses Test paths and
Data Flow Coverage web application tool for Edge Pair Test
paths and Test Cases.

11
STEPS FOR CODE-BASED TESTING
Define a FG and
number each node

Denote each node

with definition and
usage of all v and x
Find paths that
satisfy selected
coverage criteria

Create TC and TS

Run TS and kill

mutants created 12
LITERATURE REVIEW
Effectiveness of EFSM
based test selection Code coverage of EFSM
Current work
criteria (Model Based based test suites [2]
Testing) [1]

Derive mutants from Derive mutants from Derive mutants

specifications Java code from Java code

Generate Test Suites from Derive test suites from

Derive test suite
FG obtained from FG from specifications
from Java code
specifications

[1]K. El-Fakih, A. Simao, N. Jadoon and J. Maldonado, “An assessment of extended finite state machine
test selection criteria”, Journal of Systems and Software, vol. 123, pp. 106-118, 2017. 13
[2] K. El-Fakih, T. Salameh and N. Yevtushenko, “On code coverage of extended FSM based test suites:
An initial assessment”, Proc. of the 26th International Conference on Testing Software and Systems, pp.
198-204, 2014.
OBJECTIVE
 This paper evaluates code-based All-Uses test suites of
three well-known EFSM application examples and
compares their fault coverage to that achieved by
model-based All-Uses test suites.

14
CONSIDERED EFSM EXAMPLES
number:=0
Responder t3: IDISreq/DR

t2: CR/ICONind
disconnected wait
Initiator

nd
Ni
t8

O
:

IC
ID
TFTP

R/
IS
t1: IDISreq/DR

:C
t4: ICON_resp/CC

re

t7
q/
number:=0

DR
POP3 connected

t5: DT(num), DT.num==succ(number)/

AK(num)
number=succ(number)
SCP t6: DT(num), DT.num !=
succ(number)/AK(num)

15
CD player
VISUSTIN V7.10- GENERATING FLOW GRAPHS

16
DEFS-USES DERIVED FROM RESPONDER FG
Variables All defs All uses
 Total number of nodes: 1-29
IDISreq 1 4 21 24
 All Edges: {1 2}, {2 20}, {2 3},
DR 1 8 22 {3 6}, {3 4}, {4 7}, {4 5}, {7 2},
{5 2} , {5 8}, {8 9}, {9 10},{ 9
CR 1 11 2}, {10 11}, {10 12}, {12 20},
ICONind 1 7 28 {11 13}, {11 24}, {13 14}, {14
15}, { 14 24}, {15 16}, {15 17},
ICON_resp 1 5 18 {17 20}, {16 19}, {16 18}, { 19
DT1 1 16
23}, {23 29, {23 25}, {25 26},
{25 27}, {27 29}, {26 29}, {29
DT0 1 16 14}, {18 22}, {18 21}, { 22 24},
{ 21 24}, {21 14}, {24 28} , {24
CC 1 13 9 }, {28 2}
AK 1 29  Initial Node: 1
Input 1 3 15 4 5 11 19 18 21 24  Final Node: 20

return_value 1 7 8 13 29 28 22 6 7 8 12 13 16 17
22 28 29
26
num 1 19 23 29
number 1 13 27 26 23 25
TOOLS FOR GENERATING EXECUTABLE ALL USES TEST
PATHS

18
JUNIT TEST CASE
 JUnit is a unit testing framework for the Java
programming language.

 After deriving the executable test paths from the

flow graph and corresponding test suites
(input/output pairs), write test suites in JUnit.

 This JUnit test case class was executed against

the mutants.

19
CONSIDERED MUTANTS
Category Operator Description
AORB Arithmetic Operator Replacement - Binary
AORS Arithmetic Operator Replacement - Shortcut
Arithmetic
AOIS Arithmetic Operator Insertion - Shortcut

AOIU Arithmetic Operator Insertion - Unary

COR Conditional Operator Replacement

Conditional COI Conditional Operator Insertion

COD Conditional Operator Deletion

Relational ROR Relational Operator Replacement

Logical LOI Logical Operator Insertion

20
CONSIDERED MUTANTS (MUCLIPSE)

21
CONSIDERED MUTANTS (MUCLIPSE)

22
CONSIDERED MUTANTS (MUCLIPSE)

23
CONSIDERED MUTANTS (MUCLIPSE)

24
ASSESSMENT METHODOLOGY

25
ASSESSMENT METHODOLOGY (CONT.)
 Fault coverage measured by:

 𝑴𝒖𝒕𝒂𝒕𝒊𝒐𝒏_𝑺𝒄𝒐𝒓𝒆 = 𝑀𝑘𝑖𝑙𝑙𝑒𝑑 / (𝑀𝑡𝑜𝑡𝑎𝑙 – 𝑀𝑖𝑛𝑑𝑠 ) × 100

 𝑪𝒐𝒗𝒆𝒓𝒂𝒈𝒆_𝑳𝒆𝒏𝒈𝒕𝒉_𝑺𝒄𝒐𝒓𝒆
= (𝐹𝑎𝑢𝑙𝑡_𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒_𝑜𝑓_𝑇𝑆) 𝑊𝐹𝐶 + (𝑙𝑒𝑛𝑔𝑡ℎ_𝑜𝑓_𝑇𝑆) 𝑊𝐿𝑒𝑛𝑔𝑡ℎ

Given: 𝑊𝐹𝐶 = 0.99, 𝑊𝐿𝑒𝑛𝑔𝑡ℎ = 0.01

26
RESULTS AND RANKING
 Assessment of standard deviation per each mutation
operator
 Assessment of test suites per each mutation operator

 Average assessment of test suites per each mutation

operator
 Assessment of test suites per each mutation operator
category
 Average assessment of each Mutation Operator
Category

27
PROTOCOLS CONSIDERED
 Three well-known communication protocol
EFSMs were considered in this procedure:

 The Responder
 The SCP

 The POP3 specifications

28
ASSESSMENT OF AVERAGE MUTATION OPERATOR COVERAGE
PER EACH MUTATION OPERATOR (RESPONDER)

29
ASSESSMENT OF TEST SUITES PER EACH
MUTATION OPERATOR (ALL EXAMPLES)
Arithmetic Conditional Relational
All Examples
AOIS COR COI COD ROR
All-Uses Code 58.7% 47% 76% 100% 47.3%
All Uses 46.7% 31.3% 75.7% 97.7% 42.7%

30
FAULT COVERAGE ASSESSMENT FOR ALL
CONSIDERED EXAMPLES

31
RANKING OF TEST SUITE COVERAGE OF ALL
TYPES OF CONSIDERED FAULTS (COMPARE)

Ranking Mutation Score Coverage Length Score

1 All-Uses Code (63.7%) All-Uses Code (64.2%)

2 All-Uses (58.7%) All-Uses (58.60%)

32
CONCLUSION
 Code-based All-Uses test suites achieved 63.7% and 64.2 in terms of
the mutation score and coverage length score, respectively, allowing
it to slightly outperform model-based All-Uses test suites, which
attained 58.7% and 58.6 in terms of the same assessment metrics.
 Code-based and model-based All-Uses test suites provided the
highest fault coverage of 100% and 97.7%, correspondingly, in terms
mutation score for COD-based mutants, while the lowest mutation
scores of 47.3% and 42.7% were achieved when the test suites were
applied on ROR-based mutants.
 Fault coverage of conditional mutants is the highest for both test
suites, while that of rational mutants is the lowest.
 More examples need to be considered

33
 REFERENCES
[1] S. Sarraf, F. DeebaHasan and N. Garg, “A comparative study of data flow testing techniques”, International
Journal of Latest Trends in Engineering and Technology (IJLTET), vol. 2, no. 2, pp. 280-285, 2013.
[2] Guru99.com, 2016. [Online]. Available: http://www.guru99.com/static-dynamic-testing.html. [Accessed: 23-
Oct- 2016].
[3] V. Alagar and K. Periyasamy, Specification of software systems. London: Springer, 2011.
[4] A. Petrenko, S. Boroday and R. Groz, “Confirming configurations in EFSM Testing”, IEEE TSE, vol. 30, no. 1,
pp. 29-42, 2004.
[5] J. Badlaney, R. Ghatol and R. Jadhwani, "An introduction to data-flow testing", North Carolina State
University. Dept. of Computer Science, 2006.
[6] Y. Jia and M. Harman, “An Analysis and Survey of the Development of Mutation Testing”, IEEE
Transactions on Software Engineering, vol. 37, no. 5, pp. 649-678, 2011.
[7] G. Lu and H. Miao, “An Approach to Generating Test Data for EFSM Paths Considering Condition
Coverage”, Electronic Notes in Theoretical Computer Science, vol. 309, pp. 13-29, 2014.
[8] T. Shu, Z. Ding, M. Chen and J. Xia, “A heuristic transition executability analysis method for generating
EFSM-specified protocol test sequences”, Information Sciences, vol. 370-371, pp. 63-78, 2016.
[9] J. Ferrer, P. Kruse, F. Chicano and E. Alba, “Search based algorithms for test sequence generation in
functional testing”, Information and Software Technology, vol. 58, pp. 419-432, 2015.
[10] A. Pretschner, H. Lötzbeyer and J. Philipps, “Model based testing in incremental system development”,
Journal of Systems and Software, vol. 70, no. 3, pp. 315-329, 2004.
[11] A. Kalaji, R. Hierons and S. Swift, “An integrated search-based approach for automatic testing from
extended finite state machine (EFSM) models”, Information and Software Technology, vol. 53, no. 12, pp. 1297-
1318, 2011.
34
[12] P. Braione, G. Denaro, A. Mattavelli, M. Vivanti and A. Muhammad, “Software testing with code-based test
generators: data and lessons learned from a case study with an industrial software component”, Software Qual
J, vol. 22, no. 2, pp. 311-333, 2013.
 REFERENCES
[13] N. Li, U. Praphamontripong and J. Offutt, “An Experimental comparison of four unit test criteria: Mutation,
edge, all-uses and prime path coverage”, Mutation, pp. 220-229, 2009.
[14] I. Bluemke and K. Kulesza, “Reductions of Operators in Java Mutation Testing”, Proceedings of the Ninth
International Conference on Dependability and Complex Systems DepCoS-RELCOMEX. June 30 – July 4, 2014,
Brunów, Poland, pp. 93-102, 2014.
[15] J. Strug and B. Strug, “Machine Learning Approach in Mutation Testing”, Testing Software and Systems,
pp. 200-214, 2012.
[16] K. El-Fakih, A. Simao, N. Jadoon and J. Maldonado, “An assessment of extended finite state machine test
selection criteria”, Journal of Systems and Software, vol. 123, pp. 106-118, 2017.
[17] K. El-Fakih, T. Salameh and N. Yevtushenko, “On code coverage of extended FSM based test suites: An
initial assessment”, Proc. of the 26th International Conference on Testing Software and Systems, pp. 198-204,
2014.
[18] RFC1350 – The TFTP Protocol (revision 2). URL: http://www.rfc-editor.org/rfc/rfc1350.txt
[19] RFC1939, “Post office protocol - version 3,” 1996.
[20] D. Hogrefe, OSI formal specification case study: the Inres protocol and service, Bern: Switzerland, 1992.
[21] W. H. Chen, “Executable test sequences for the protocol data flow property”, Int. Conference on Formal
Techniques for Networked and Distributed Systems, pp. 285-299, 2011.
[22] D. Seifert, Conformance testing based on uml state machines. In Formal Methods and Software Engineering,
London: Springer, 2008.
[23] Y. S. Ma, J. Offutt and Y. R. Kwon: “MuJava: An automated class mutation system”, JSTVR, vol. 15, no. 2,
pp. 97-133, 2005.
[24] "Visustin - Flow chart generator", Aivosto.com, 2016. [Online]. Available: 35
http://www.aivosto.com/visustin.html. [Accessed: 23- Oct- 2016].
[25] M. Hassoun, "Fault Coverage and Diagnosis of Protocols and Systems Modeled As Extended Finite State
Machines," M.S. Thesis, Dept. Computer Science and Eng., American University of Sharjah, 2015.
THANK YOU

36