Test Derivation for SDN-Enabled Switches: A Logic Circuit Based Approach

Abstract

The paper is devoted to testing critical Software Defined Networking (SDN) components and in particular, SDN-enabled switches. A switch can be seen as a forwarding device with a set of configured rules and thus, can be modelled and analyzed as a ‘stateless’ system. Correspondingly, in this paper we propose to use appropriate logic circuits or networks to model the switch behavior. Both active and passive testing modes can benefit from such representation. First, this allows applying well-known test generation strategies such as for example, test derivation techniques targeting Single Stuck-at Faults (SSFs). We also specify a number of mutation operators for switch rules and propose an algorithm for eliminating equivalent mutants via SAT solving. Logic circuits simulating the behavior of the switches can be effectively utilized for run-time verification, and such logic circuit based approach is also discussed in the paper. Preliminary experimental results with Open vSwitch, on one hand, demonstrate the necessity of considering new fault models for logic circuits (apart from, for example well established SSFs) and on the other hand, confirm the efficiency of the proposed test generation and verification techniques.

1 Introduction

As communication technologies progress rapidly, more attention is now paid to virtual networks and various ‘softwarization’ techniques. In fact, virtualization opens a large number of possibilities when the common resources can be effectively shared and specific network services manage the policies for using these resources. Nevertheless, whenever a given virtual network is requested, it is necessary to assure that its implementation strictly corresponds to the submitted request. In order to guarantee this property, all the hardware and software components involved into the chaining and virtualization processes need to be thoroughly tested and verified.

Software Defined Networking [20] is known to be one of the fundamental building blocks for the creation of virtual networks and thus, SDN-enabled devices such as for example, switches and/or controllers need to be certified, guaranteeing the absence of (specific) bugs and misconfigurations.

We focus on testing SDN switches that act as forwarding devices receiving and sending network packets in accordance with a set of configured rules. Nowadays, such devices are predominantly implemented in software, and therefore different software bugs can induce different functional faults. For example, while using the ONOS controller [2] and Open vSwitch (OVS) [22] we have detected a potential overflow with respect to (w.r.t.) the switch port numbers. Namely, any request with an output port number which is greater than or equal to $2^{16}$ produces inconsistent results. Each of such requests gets the assigned port number modulus $2^{16}$. The OpenFlow Switch Specification [18] states that the maximal physical and logical port number is 4294967040 (0xffffff00). Therefore, one can conclude there is a bug in the OVS implementation¹. Such software bugs lead to the incorrect packet processing, i.e., the specification given as a set of rules for the switch is not respected. Thus, detecting such bugs is crucial for improving the functional correctness of the SDN infrastructure.

In this paper, we apply model based testing techniques for an SDN-enabled switch. In particular, we propose to model the switch behavior as a corresponding logic circuit in order to take advantage of various scalable manipulations over such circuits as well as to benefit from well-established techniques for their testing. Both, active (Sects. 4 and 5.1) and passive testing (Sects. 4.5 and 5.2), can take advantage of such representation. In particular, we estimate the usefulness of logic circuit based fault models such as for example Single Stuck-at Faults (SSFs) for detecting bugs and misconfigurations in the SDN-enabled switch implementations. Correspondingly, we introduce potential mutations over the switch rules and discover which of these mutations can be effectively detected using the proposed logic circuit based approach. Moreover, we discuss how Boolean Satisfiability (SAT) solvers can be utilized for detecting equivalent mutants. Finally, we propose a scalable solution for the switch monitoring on the basis of logic circuits and related operations.

The paper is organized as follows. Section 2 contains the necessary background and notations. Section 3 summarizes the related work in the area of SDN switch testing and verification. The logic circuit based approach for switch testing is presented in Sect. 4. Preliminary experimental results for a set of switch rules are shown in Sect. 5. Section 6 concludes the paper.

2 Background and Notations

2.1 Software Defined Networking

In Software Defined Networking architectures, the control plane is decoupled from the forwarding (data) plane [20], i.e., a logically centralized control function maintains the state of the network and provides instructions to the data plane (packet forwarding devices). Switches located in the data plane then forward data packets according to these control instructions, more specifically, forwarding and filtering rules [23]. Each switch should process the received packets according to the installed forwarding rules. A forwarding rule consists of three parts: a packet matching part, an action part, and a location/priority part. The matching part describes the values a received network packet should have in order for the rule to be applied (to the packet). The action part indicates how to process the matched network packets; the location/priority part controls the rule hierarchy using tables and priorities. In SDN, to steer network packets in the data plane, forwarding rules are ‘pushed’ to the switches by the controller that has a global view of the network; switches are configured remotely and dynamically through interfaces using protocols such as the OpenFlow (OF) protocol [16]. The forwarding rules are grouped in different flow tables and are considered to be the configurations of switches with respect to packets and application flow management.

As an example of rules installed in a switch, consider the set of rules defined in Table 1. The table includes the following matching parameters: Flow Table, a virtual partition for the installed rules; Priority, the order attributed to the rule to be applied with respect to other rules in the flow table; Input Port (In_port), the ingress port of the incoming packets; Ethernet Type (Eth_type), the type of traffic carried by the Ethernet datagram; Source and Destination IP Addresses respectively (IP_source, IP_dest), that define the IP protocol source and destination addresses. The output ports (Output) defines the set of ports where a matching packet should be forwarded.

Table 1. Switch example

For example, according to Table 1, the third rule is specified in the flow table 1 with the priority 501. When the packets having the source IP address 10.0.0.1 and destination IP address 10.0.0.2 arrive to Port 1 of the switch, these packets have to be forwarded via the (output) Port 2 of the switch.

2.2 Logic Circuits and Related Fault Models

A combinational circuit is composed of logic gates; each logic gate implements a Boolean function. Sequential circuits include on one hand, a combinational logic, and on the other, memory elements, namely latches. In this paper, we consider circuits ‘without memory’, i.e., combinational circuits where the circuit output significantly depends only on its current input.

Following [12], we consider three types of faults that can occur in the circuit implementation, namely Single Stuck-At Faults, Single Bridge Faults, and Hardly Detectable Faults. A circuit that contains a fault is referred to as a mutant, as usual.

• Single Stuck-At Fault (SSF) Mutants occur when one circuit gate gets ‘stuck’ at a given logical value (“1” or “0”);

• Single Bridge Fault (SBF) Mutants occur when a given input of a given logical gate is wrongly wired (bridged), i.e., taking the input from the wrong gate;

• Hardly Detectable Fault (HDF) Mutants occur when a single gate changes its output for a single input.

A number of test generation strategies against the faults listed above have been proposed in the last decades. The interested reader can, for example refer to [15, 21]. Moreover, test suites derived against the SSFs are claimed to have high fault coverage with respect to other types of circuit mutants. In this paper, we investigate the fault coverage of these techniques when testing SDN-enabled switches effectively described by corresponding logic circuits.

2.3 Notations

A rule R defined in the switch configuration is represented by the following implication: $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_i \in V_i \; \& \; \ldots \; \& \; p_n \in V_n) \implies output\_ports = \{o_1, o_2, \dots , o_m\}$. In this case, $p_i$ refers to an input parameter, $o_i$ refers to an output port and the sets $V_1$, $V_2$, $\dots$, $V_n$ define a range or an interval for each switch parameter $p_1$, $p_2$, $\dots$, $p_n$, correspondingly². For convenience, we denote as $\varPi$ the projection operator, characterized with a parameter $p_i$ such that for a given rule $R=(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_i \in V_i \; \& \; \ldots \; \& \; p_n \in V_n) \implies output\_ports = \{o_1, o_2, \dots , o_m\}$, $\varPi _{p_i}=V_i$. Similarly, we denote the output projection of R as $\varPi _{out}=\{o_1, o_2, \dots , o_m\}$.

An output mutant for the rule R is defined as follows: $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_i \in V_i \; \& \; \ldots \; \& \; p_n \in V_n) \implies output\_ports = \{o{'}_{1}, \ldots , o{'}_{m'}\}$, and $\{o{'}_{1}, \ldots , o{'}_{m'}\} \not = \{o_1, \dots , o_m\}$.

A parameter value mutant for the rule R is defined as follows: $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_i \in V'_i \; \& \; \dots \; \& \; p_n \in V_n) \implies output\_ports = \{o_1, o_2, \dots , o_m\}$, and $V'_i \not = V_i$.

In the example illustrated in Table 1, the number of parameters is $n = 6$, and $|V_i| = 1, i \in \{1, \dots , 6\}$. Note that $m=3$ (the number of output ports) for the rules in Table 1.

3 Related Work

A number of publications and online presentations have been previously devoted to the verification and testing of SDN switches. In particular, a number of works are dedicated to the validation of the consistency of the switch rules [1, 9].

There exist a large number of publications that concern the overall verification of the data plane. Different techniques can be applied in this case, for instance, Header Space Analysis of network packets [10] or Model Checking and Symbolic Execution [6]. Several properties for the whole data plane can be checked in this case, such as reachability issues, absence of loops/holes, etc.

Data plane analysis has also been performed in an active mode. In this case, the packets to be sent through the switches have been specifically generated to capture various network failures. The approaches of this set mostly cover automatic network traffic or flow generation (see, for example [5, 7, 25]). We however note that the analysis is usually performed for checking the paths/networks implemented in the data plane rather than checking the functionality of a given critical forwarding device.

Existing works on testing a given switch can be mostly divided into two groups. The approaches of the first group have a number of pre-defined test purposes that either cover some stress situations for the switch (such as, for example, a flow table capacity) or verify given scenarios formally described in the OpenFlow specification [19]. The approaches of the second group rely on formal models. In particular, in [24] so called Pipelined Extended Finite State Machines are introduced for describing a switch behavior. Note that in this case, a switch is considered as a stateful system which immediately complicates the process of test generation and execution.

We also note that the idea of expressing a switch or a composition of those with the use of Boolean algebra or a system of (partially specified) Boolean functions has been employed before. For example, in [1] a set of switch rules is described by a corresponding Binary Decision Diagram that later is used for verifying their consistency. Moreover, a set of Boolean expressions have been constructed in [14] for the data plane verification based on the SAT problem.

However, to the best of our knowledge, there does not exist an approach which proposes the use of Boolean satisfiability or related logic circuits/networks for deriving active test suites with guaranteed fault coverage. Moreover, ‘classical’ logic circuit based testing strategies such as for example, SSF detection, have not been previously used for checking the functionality of SDN-enabled switches. Likewise, the authors are not aware of any works where logic circuits and their software and/or hardware implementations are used for a passive testing/monitoring of an SDN-enabled switch behavior.

4 Logic Circuit Based Approach for Test Suite Generation

4.1 Introducing a Fault Model

We assume that the switch implementation has no faults if each packet is processed exactly in the way that the switch configuration requires. Moreover, if for a given packet pkt there is no rule R in the switch configuration such that the matching part $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n)$ contains the necessary preamble, the packet pkt is simply dropped by the switch, i.e., should not be forwarded anywhere. We note however that a switch can output the packet to ‘consult’ with the SDN controller about the action applied to an ‘unknown’ packet [16]; the controller might alter the rules in the switch configuration as a result. We assume this is a different specification; further, in this work, we do not focus on such case. The proposed fault model has three items, as usual, namely $FM = \langle S, =, FD\rangle$ where S, the specification, is the set of switch rules, i.e., the rule forwarding configuration of the switch (referred in this paper simply as switch configuration); $=$ is the conformance relation represented by the equality, and FD is the fault domain where the potential switch implementations are explicitly enumerated. As usual, we are interested in deriving exhaustive test suites, such that $\forall I \in FD, I \ne S$, is detected by the test suite.

We also note that the system specification in this case can be complete (completely specified) or partial. The set S of switch rules is said to be complete if for each preamble $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n)$ there exists a rule $R \in S$ such that $R = ((p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n) \implies output\_ports = \{o_1, o_2, \dots , o_m\})$. Otherwise, the specification S in the fault model FM is partial. We further discuss how completeness and partiality of S affect the exhaustiveness of the test suites derived using well-known logic circuit based fault models.

4.2 Deriving a Logic Circuit for a Switch Specification

The specification S represented by a set of switch rules is not scalable for solving different problems, such as for example, searching for two rules in possibly different tables that coincide or that on the contrary, contradict each other. We therefore, propose to build a logic circuit that preserves the behavior of S on one hand, but allows taking advantage of several scalable manipulations over the Boolean vectors (logic circuits) on the other hand. Such logic circuit LC can be derived in different ways and in this work, we focus on a somehow straightforward approach for this purpose, i.e., we propose to use logic synthesis solutions from a Look-up-Table (LUT) for a system of (partially specified) Boolean functions³. The corresponding procedure is described in Algorithm 1.

For the running example of the set of switch rules listed in Table 1, the LUT derived by Algorithm 1 has four lines illustrated in Table 2. Note that dashes (−) denote ‘don’t care’ terms⁴.

Table 2. LUT for the switch running example

4.3 Test Suite Generation

Once a logic circuit LC that simulates the behavior of the switch with the rules S is derived, one can apply different techniques for test generation. On one hand, ‘classical’ logic circuit testing strategies such as for example, test derivation techniques for SSF detection can be exploited. In this case, the circuit LC is mutated in order to obtain a set of mutants of different kinds, such as for example SSF or HDF. The test suite derived to kill each mutant of a particular type can later be applied to the implementation of an SDN switch. The goal of deriving such test suite is to distinguish the output of a correct implementation from an assumed incorrect implementation (mutant)⁵. The advantage of this approach is that logic circuit testing techniques are well studied and elaborated and there exist a number of tools for such test derivation. In this work, we used the tool developed in [12] together with the logic synthesis and verification tool called ABC [4]. Preliminary experimental results for such test generation strategy are presented in Sect. 5.

On the other hand, in some cases, certain properties for a test suite fault coverage can be guaranteed. For example, for SSF mutants the following statements hold.

Proposition 1

If S is complete and $\exists \; i \in \{1, \dots m\}$ such that $\exists !\; R \in S$, $R = ((p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n) \implies output\_ports = \{o_1, o_2, \dots , o_m\})$ and $o_i \notin \{o_1, o_2, \dots , o_m\}$, then each output fault in the rule R is detected by an exhaustive test suite w.r.t. SSFs.

Proof

The completeness of the specification S automatically implies the completeness of the system of Boolean functions implemented by the circuit LC (Algorithm 1). Each output fault $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n) \implies output\_ports = \{o'_1, o'_2,\ldots ,o'_{m'}\}$, and $\{o'_1, o'_2,\ldots ,o'_{m'}\} \not = \{o_1, o_2, \dots , o_m\}$ is only detected by a test suite if this test suite includes the input pattern $B_1 B_2 \dots B_n$ that corresponds to the rule R. As the test suite TS contains a pattern that distinguishes each SSF mutant of the logic circuit LC, for the output i of LC this pattern can only be $B_1 B_2 \dots B_n$, otherwise the stuck-at-one fault in the i-th output cannot be detected (due to the uniqueness of the rule R). $\square$

Note that whenever the set S of switch rules is not complete, the logic circuit LC is derived for a system of partially specified Boolean functions. Therefore, the behavior of the circuit over the undefined patterns can be specified in different ways. In our approach and in our experiments, we use ABC, which sets the corresponding outputs to 0. This fact allows to guarantee the fault coverage for output mutants of the rules when initially the specification S is not complete.

Proposition 2

If for a set of rules S $\exists \; i \in \{1, \dots m\}$ such that $\exists !\; R \in S$, $R = ((p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n) \implies output\_ports = \{o_1, o_2, \dots , o_m\})$ and $o_i \in \{o_1, o_2, \dots , o_m\}$, then each output fault in the rule R is detected by an exhaustive test suite w.r.t. SSFs.

Proof

Similar to Proposition 1, a test suite TS which detects each stuck-at-zero fault on the i-th output of LC must contain a pattern $B_1 B_2 \dots B_n$ that corresponds to the preamble $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n)$ of rule R. This exact pattern detects each output fault in the rule R.

Corollary 1

If in the set S of switch rules, each output port is used in at most one rule, then an exhaustive test suite w.r.t. SSFs is also exhaustive w.r.t. rule output mutants.

We note however, that the above statements do not necessarily hold for the parameter value mutations. Such faults can in some cases be detected by other mutants of logic circuits such as bridges or hardly detectable faults. Nevertheless, thorough investigation of the correlation between the mutations of rules and those of logic circuit still needs to be performed. Such investigation is left for future work.

4.4 SAT Solving for Equivalent Mutant Detection

Whenever possible rule mutations are enumerated explicitly and therefore, a test suite TS is derived under the White Box testing assumption aiming at killing all the mutants of certain type, the question of equivalent mutants automatically rises [8]. Indeed, mutations of different orders (especially second and higher) have a high probability of deriving an equivalent mutant. However, as the number of patterns can be rather high ($2^{\sum _{i=1}^{n}\lceil log_2(1+max(\bigcup _{R\in S}\varPi _{p_i})) \rceil }$), applying/checking all such patterns can be a time consuming task, and thus, detecting equivalent mutants by direct (brute force) search becomes unfeasible.

Correspondingly, such equivalent mutants can be effectively detected whenever two logic circuits LC and $LC_M$ for both the specification S and the mutant M under investigation, are derived. Indeed, the equivalence decision problem can be reduced to the well-known SAT problem. For this reason, a miter of two circuits can be derived. For two logic circuits LC and $LC_M$ with the set $X = \{x_1, \dots , x_k\}$ of inputs and the sets $O = \{o_1, \dots , o_p\}$ and $O' = \{o'_1, \dots , o'_p\}$ of outputs, a miter Mit with the set $X = \{x_1, \dots , x_k\}$ of inputs and a single output is derived as follows. The output function of Mit is the result of a logic OR operation of the functions $f_1, \dots , f_p$ that are implemented as the XORs of output functions $g_1, \dots , g_p$ and $h_1, \dots , h_p$ of the circuits LC and $LC_M$ correspondingly, i.e., $f_j = g_j \oplus h_j$, $j \in \{1, 2, \dots , p\}$. Circuits LC and $LC_M$ are equivalent, and so are the sets of rules S and M, if each output of the miter Mit always equals 0, i.e., when the corresponding Boolean function is UNSAT. Algorithm 2 implements this strategy to detect equivalent mutants.

The correctness of the proposed equivalence check is established by the following proposition.

Proposition 3

For a given set S of switch rules and a given mutant M of this specification, Algorithm 2 returns a test case killing M if and only if the mutant M is not equivalent to S.

Proof

Indeed, the circuit Mit implements a constant 0 if and only if the outputs $B\_port$ coincide for all input patterns $B_1 B_2 \dots B_n$ (Algorithm 1). Thus, a satisfying pattern B for the function f returns an input $B_1 B_2 \dots B_n$ for the preamble $(p_1 \in V_1 \; \& \; p_2 \in V_2 \; \& \; \dots \; \& \; p_n \in V_n)$ where the output ports differ.

We note that such equivalence check can be performed over the logic circuit representations in a scalable way. The reason is that both circuits LC and $LC_M$ are combinational, i.e., without latches or internal memory. For sequential circuits, the derivation of the miter as well as the SAT problem formulation is in fact much more complex. The latter means, that if modelling a switch as a stateful system, for example when taking into account its potential communication with an SDN controller, the solution of the equivalence check might not be scalable. More research and experiments are needed in this area, and these tasks are left for the future work.

4.5 Logic Circuits for Switch Monitoring

We previously discussed the use of logic circuits for active test generation for an SDN switch. In fact, whenever the access to the switch is limited and its behavior can only be observed, a logic circuit LC modelling the specification S can still be effectively utilized. The reason is that the simulation of LC can in some cases be much faster than the search of the particular switch rule and its further application to conclude about the expected output port(-s). In other words, the task of the switch monitoring that verifies that the packets are forwarded to the exact ports specified by S, can be reduced to the problem of the circuit simulation⁶, i.e., obtaining an output pattern for a given input (pattern). This approach is described in Algorithm 3.

As discussed in Sect. 5, in many cases, the verdict about the correct or incorrect application of a given switch rule can be made much faster when the logic circuit representation is exploited.

5 Preliminary Experimental Results for Open vSwitch

The experiments have been performed on the widely-known SDN-enabled switch, Open vSwitch (OVS) [22] version 2.0.2. To simulate a ‘close-to-real’ switch behavior, the popular Mininet [17] tool was utilized; Mininet provides an easy way to simulate and prototype SDN networks (using OVS). The topology of the simulated network is similar to the one presented in our previous work [3] and shown in Fig. 1. This topology models the data plane as a graph where all switches are connected to an SDN controller. In Fig. 1, hosts and switches are labeled with strings starting with the letters h and s respectively. For each edge in the graph, the corresponding port number used by the two nodes is depicted; for example, in the edge $(s_2,s_3)$, the label ‘33’ indicates that the port 3 at switch $s_2$ is connected to the port 3 at switch $s_3$. In addition, the Ethernet MAC addresses for each of the hosts are shown above or below each host. For our experiments, without loss of generality, we chose the switch $s_3$ (depicted with a dotted pattern) as the system under test. The ONOS [2] controller version 1.10.4 was used for all the experiments. The experiments were executed under different virtual machines running under a VirtualBox Version 5.2.8 r121009 for Mac OS X 10.13.4. The virtual machines have the characteristics shown in Table 3.

Fig. 1.

Experimental setup topology

Table 3. Experimental platform

5.1 Logic Circuit Fault Models for SDN-Enabled Switch Fault Model

There exist a large number of possibilities how an SDN network can be (re-) configured, and therefore how to obtain the switch rule set specification S. The SDN controller by itself is not ‘responsible’ for this configuration. In an SDN architecture, applications query the controller for information of the data plane and request the implementation of different rule sets for the forwarding devices (e.g., SDN-enabled switches). The applications can have different goals; for example, one application can monitor the data plane in order to balance the load between the network links (edges). One of the most common applications for an SDN network is the Layer 2 switching application; this application forwards Layer 2 packets (e.g., Ethernet) between hosts using the shortest paths.

To provide meaningful results, we programmed the data plane with a Layer 2 switching application. The rules were pushed through the controller using the REST interface. As an example, one rule $R\in S$ installed on $s_1$ is the following: $R = ((\text {INPUT}\_\text {PORT} \in \{1\} \& \text {ETH}\_\text {SRC} \in \{\text {9a:d8:73:d8:90:6a}\} \& \text { ETH}\_\text {DST} \in \{\text {ff:ff:ff:ff:ff:ff}\} \implies output\_ports = \{2,3\})$. After the rules were pushed, communication from / to any host was successfully achieved via the Ethernet protocol.

A logic circuit LC was derived from the specification S using Algorithm 1. The logic synthesis tool used for deriving LC was ABC [4]. After the synthesis, LC has 99 inputs and 4 gates. The logic circuit was then saved into the Berkeley Logic Interchange Format (BLIF), and a BLIF Mutant Generator (BMG) [12] tool was executed to generate mutants for the SSF, SBF, and HDF types. The total number of mutants is 214, where 206 mutants are all SSF mutants, 4 are randomly chosen SBF, and likewise 4 are randomly chosen HDF mutants. A distinguishing pattern was found for each non-equivalent mutant. For each of the fault models, a test suite was obtained, i.e., $TS_{SSF}$, $TS_{SBF}$, and $TS_{HDF}$; furthermore, the union of all 3 test suites was used to obtain $TS_{ACF}$, a test suite for all circuit faults⁷. The original BLIF circuit contains a sum of products, hence, only 4 gates (the output gates). In order to check if the fault coverage increases with different circuit representations, the original BLIF file was re-synthesized as an AND-INVERTER graph (AIG), we hereafter refer to this circuit as $S'$ (functionally equivalent to S). $S'$ has 99 inputs and 395 gates, and therefore, the total number of mutants is 1778, where 998 mutants are all SSF mutants, 395 are randomly chosen SBF, and likewise 395 are randomly chosen HBF mutants. The same procedure was performed on $S'$ to obtain the corresponding test suites.

To check the fault coverage of traditional digital circuit fault models, a set $\mathcal {M}$ of 45 mutants of S was generated. The set of mutants contains different order mutants. After running Algorithm 2 to remove from $\mathcal {M}$ the equivalent mutants (1 was removed), each pattern p in each of the test suites $TS_{SSF}$, $TS_{SBF}$, $TS_{HDF}$, $TS_{ACF}$ was used to simulate the behavior of S and compare it to the behavior of each mutant M in the non-equivalent mutant set $\forall M\in \mathcal {M}$. The fault coverage⁸ obtained for each of the test suites is shown in Table 4.

Table 4. Fault coverage for traditional digital circuit fault models

As seen in Table 4, the fault coverage of traditional logic circuit fault models reaches $100\%$ for a Layer 2 switching specification. Therefore, we conclude that test suites derived based on traditional logic circuit fault models have a high fault coverage for SDN-enabled switch faults. An interesting aspect is that the fault coverage highly increases when the original circuit specification is transformed into an AIG. It is reasonable to assume that AIGs have more gates, and therefore more mutants and distinguishing patterns are obtained with such representations. Thus, different functional errors in the switch rules can be covered by a larger test suite when derived based on such AIGs.

5.2 Using Logic Circuits for Monitoring

When monitoring functional properties of a given IUT $\mathcal {I}$, there are two modes of operation (off-line and on-line/run-time). As discussed before, run-time monitoring requires that the monitor (in this case LC) is not slower than $\mathcal {I}$. A large body of studies has been dedicated to the proposal of fast monitoring algorithms (see, for example [13]) and heuristics (see, for example [11]) to enhance the performance of monitoring solutions. It is presumable that a combinational circuit has a constant (or near to constant) computational time for any input pattern. On the other hand, switch implementations such as Open vSwitch effectively work by caching the corresponding actions to be applied to a matching packet, and thus, these cached actions are applied to subsequent packets matching the rule [22]. The match lookup (and corresponding action to be applied) is done using a set of hash tables which are increased with each unique match [22]. Therefore, a priori the simulation of a logic circuit for the implementation of the switch behavior should be faster than the described caching mechanism.

To verify that logic circuits are indeed suitable for run-time monitoring of SDN-enabled switches, we performed the following experiment. On one hand, 10000 rules were pushed into the switch $s_3$ (IUT) using the ONOS controller. On the other hand, a logic circuit was obtained using Algorithm 1 for the specification containing the same 10000 rules installed in the IUT. The time Open vSwitch takes to process one packet and the time taken to simulate a single pattern of the logic circuit are measured as follows. To determine the time to process one packet, monitors were installed on each switch port (interface). The time difference between the packet ingress and the packet egress was measured to be $\sim$0.29 ms, executing Open vSwitch under VM2. As the time to simulate a single pattern in ABC (given a synthesized circuit) is considerably low, precision issues may occur while measuring the time to simulate a single pattern. Furthermore, reading the file and writing the response to the standard output take most of the simulation time. For that reason, the relevant values were extracted from the packet used, and the Open vSwitch was simulated one thousand times. The time taken to simulate a single pattern was measured to be $\sim$0.003408 ms.

Discussion. As a conclusion of the presented experiments, it can be seen that the logic circuit simulation is more than 85 times faster (compared to the switch packet processing). Arguably, the difference in the input/output interfaces for the different environments (packet input in a switch vs. file read in a circuit simulation) can also affect the time estimations. However, capturing packets at a network interface is done when the packet has been processed by the interface and processed by the operating system, therefore, reading a packet is done from the internal (RAM) memory of the devices. On the other hand, the file used to simulate a pattern in the circuit simulation is performed from a hard drive (HD). For that reason, it is reasonable to assume the time measurements performed over the switch have an inherent advantage (RAM access is much faster than HD access). Therefore, theoretically, the speed-up may be even larger when considering the same input/output interfaces.

It is of special interest to accurately estimate the obtained speed-up due to its potential applications not only for testing reasons but, for optimizing the switch implementations. Performing such investigation of the speed-up obtained via the logic circuit representation of a set of rules is left for the future work.

6 Conclusion

In this paper, we proposed a logic circuit based approach for testing SDN-enabled devices. It allows to take advantage of well established test generation strategies for logic circuits as well as of scalable manipulation over Boolean vectors and functions. We also introduced some mutation operators over the switch rules and discussed how logic circuit and related SAT solving can be utilized for detecting equivalent mutants. Finally, we considered run-time verification of switches and investigated the use of logic circuits in this case. Preliminary experiments with Open vSwitch confirm the effectiveness of the proposed approaches.

As future work, we foresee a number of directions. First, as the main focus of the paper is on test case derivation, important aspects of test execution are left for future work. Additionally, we plan to thoroughly investigate the correlation between the mutants of the switch rules and those of logic circuits. At the same time, it is interesting to analyze which kind of bugs and inconsistencies in an SDN controller or a controller-to-switch communication can be detected via proposed ‘stateless’ approach. Equivalent mutants for switches and the corresponding SAT solving solution also need further investigation. Likewise, it is interesting to study related complexity issues in this case to see if the SAT solving can scale in a general case or specific classes over switch rules should be defined. Finally, we would like to perform more experiments on other real SDN infrastructures and environments with software and/or hardware implementations for checking the effectiveness of logic circuit solutions.