Bounded Verification of Multi-threaded Programs via Lazy Sequentialization

This article builds on and extends our previous work, where we introduced the translation and gave preliminary experimental results [52] and described the tool framework [51]. Here, we now give a more detailed description of the approach (including a formal definition of the sequentialization transformation and formal correctness proofs) and a more in-depth evaluation, including hard benchmarks that cannot be solved by any other approach.

In summary, in this article, we make the following contributions:

The remainder of the article is organized as follows: We first introduce the syntax and the semantics of multi-threaded and bounded multi-threaded programs in Section 2. We give a detailed description of our translation, along with an informal correctness argument, and formalize this translation as a set of rewrite rules in Section 3. We prove its correctness in Section 4. We describe Lazy-CSeq’s implementation in Section 5 and summarize our experimental evaluation in Section 6. We discuss related and future work in Sections 7 and 8, respectively and end with our conclusions in Section 9.

The syntax of multi-threaded programs is defined by the grammar shown in Figure 1. Terminal symbols are set in typewriter font. The notation \(\langle \mathit {n}\ \texttt {t}\rangle ^*\) represents a possibly empty list of non-terminals n that are separated by terminals t; \(\mathit {id}\) denotes a generic variable, x a local variable, y a shared variable, m a mutex, t a thread variable, p a procedure name, and l a label. We assume expressions e to be formed over local variables, Boolean literals true and false, and integer literals, which can be combined in a type-correct way using the standard Boolean and mathematical operators and the program’s side-effect free functions. We adopt the usual square-bracket notation for arrays to indicate elements of fixed-sized vectors of scalar variables. We denote Boolean expressions by b.

A multi-threaded program P (see Figure 2(a) for an example) consists of a list of global variable declarations (i.e., shared variables), followed by a list of procedures. Shared variables are always initialized to a type-specific default value. We denote the default value by 0, regardless of the type. Each procedure has a list of zero or more typed parameters, and its body has a declaration of local variables followed by a statement. A local variable can be declared as static, as in C. Such static variables are also initialized to 0. A statement is either a sequential, a concurrent, or a compound (i.e., a sequence of statements enclosed in braces) statement.

A sequential statement \(\mathit {seq}\) can be an assume- or assert-statement, an assignment, a call to a procedure that takes multiple parameters (with an implicit call-by-reference parameter passing semantics), a return-statement, a conditional statement, a while-loop, a goto- or a skip-statement with a non-numerical label l. Note that by construction all jump targets are skip-statements, because these are the only statements where we allow labels.

Sequential statements affect only the thread-local control flow and involve only local variables; the sequentialization thus does not need to simulate context switches at these [75]. Unlike global variables, local variables remain uninitialized after their declaration; therefore, until explicitly set by an appropriate assignment statement, they can nondeterministically assume any value allowed by their type. We also use the symbol * to denote the expression that nondeterministically evaluates to any possible value; hence, with x:= *, we mean that x is assigned any possible value of its type domain.

A concurrent statement \(\mathit {conc}\) can be a concurrent assignment, a call to a thread routine, such as a thread creation or join, or a mutex operation (i.e., init, lock, unlock, and destroy). A concurrent assignment assigns a shared (respectively, local) variable to a local (respectively, shared) one. A thread creation statement \(t \texttt {: = create} p(e_1,\ldots ,e_n)\) spawns a new thread from procedure p with expressions \(e_1,\ldots ,e_n\) as arguments. A thread join statement, \({\tt join}\) t, suspends the enabled thread until the thread identified by t terminates its execution, i.e., after the thread has executed its last statement. Lock and unlock statements, respectively, acquire and release a mutex. If the mutex is already acquired, then the lock operation is blocking for the thread, i.e., the thread is suspended until the mutex is released and can then be acquired.

We assume that a valid program P satisfies the usual well-formedness and type-correctness conditions. We also assume that P contains a procedure main, which is the starting procedure of the only thread that exists in the beginning. We call this the main thread. We further assume that there are no calls to main in P and that no other thread can be created that uses main as starting procedure.

A thread configuration of a multi-threaded program P is a triple \(\langle \mathit {locals}, \mathit {pc}, \mathit {stack}\rangle\), where \(\mathit {locals}\) is a valuation of the thread’s local variables, \(\mathit {pc}\) is the program counter that points to the next statement \(P(\mathit {pc})\) in P to be executed, and \(\mathit {stack}\) is a stack of procedure calls. A thread configuration is initial if \(\mathit {locals}\) is any valuation that assigns 0 to each static variable and an arbitrary value otherwise, \(\mathit {pc}\) is the program counter of the first statement of the thread, and \(\mathit {stack}\) is the empty stack (denoted by \(\perp\) in the following). At a procedure call, the program counter of the caller and the current valuation of its local variables are pushed onto the stack, and the control moves to the initial location of the callee. At a procedure return, the top element of the stack is popped, and the local variables and the program counter are restored. Any other statement follows a standard C-like semantics.

A thread identifier is a non-negative integer. A multi-threaded program configuration c consisting of n active threads with identifiers \(I=\lbrace i_1, \dots , i_n\rbrace\) is a tuple of the form \(\langle \mathit {sh},\mathit {en},\lbrace \mathit {th}_i\rbrace _{i\in I}\rangle\), where (1) \(\mathit {sh}\) is a valuation of the shared variables, (2) \(\mathit {en}\in I\cup \lbrace \dagger \rbrace\) is either the identifier \(i\in I\) of the enabled thread (i.e., the only active thread that is allowed to make a transition), or the token \(\dagger\) to mark error configurations, and (3) \(\mathit {th}_i\) is the configuration of the thread with identifier i. A configuration \(c=\langle \mathit {sh},\mathit {en},A\rangle\) is initial if \(\mathit {sh}\) is the default evaluation of the shared variables, \(\mathit {en}=0\) is the identifier of the main thread, and \(A=\lbrace \mathit {th}_0\rbrace\), where \(\mathit {th}_0\) is an initial configuration of main (i.e., the main thread); c denotes normal termination if \(\mathit {en}\ne \dagger\) and \(A=\emptyset\).

A transition of a multi-threaded program P from a configuration c to a configuration \(c^{\prime }\), denoted by \(c\rightarrow ^{i}_{P}c^{\prime }\), corresponds to the execution of a statement by the enabled thread (i.e., the thread with identifier \(i=\mathit {en}\)). If the statement being executed is sequential, then only the configuration \(\mathit {th}_\mathit {en}\) of the enabled thread is updated, as usual. The execution of an assert-statement with a condition that evaluates to false causes a transition into an error configuration, from which no further transitions can be taken. The execution of an assume-statement allows a transition only if the condition evaluates to true. The execution of a create-statement adds (with a fresh identifier i) a new thread configuration to the configuration of the multi-threaded program, which consists of a valuation of the local variables the thread start function, the program counter pointing to its initial statement, and the empty stack. The execution of a join-statement allows a transition for the enabled thread only if the thread identified by the statement’s argument t has already terminated its execution (i.e., \(t\notin I\)). The execution of a lock-statement allows a transition for the enabled thread only if the mutex m is not acquired by any thread; it leads to a new configuration where the value of m is set to \(\mathit {en}\). The execution of an unlock-statement on a mutex m, held by the enabled thread, frees it. When the enabled thread terminates, its configuration \(\mathit {th}_{\mathit {en}}\) is removed from the program configuration. The enabled thread in \(c^{\prime }\) is nondeterministically selected from the pool of active threads of \(c^{\prime }\). Note that for any schedule for P under which the execution of a join- and lock-statement blocks there is an equivalent execution with a different schedule under which the statement does not block. For example, any execution of P containing a join from a thread \(t_1\) on a thread t₂ can always be captured by another execution where t₁ is pre-empted immediately before the join and is re-scheduled only after t₂ terminates (if at all). This re-scheduling does not affect the reachability of error states, which is what our approach is checking for. For the purpose of our proofs, we therefore assume without loss of generality that join- and lock-statements never block in an execution. We finally define \(\rightarrow ^{}_{P}\) to be the union of all relations \(\rightarrow ^{i}_{P}\).

Let P be a multi-threaded program with configurations c and \(c^{\prime }\). A run or execution of P from c to \(c^{\prime }\), denoted by \(c\leadsto ^{}_{P}c^{\prime }\), is any sequence of zero or more transitions \(c_0 \rightarrow ^{}_{P} c_1\rightarrow ^{}_{P}\cdots \rightarrow ^{}_{P} c_n\) where \(c=c_0\) and \(c^{\prime }=c_n\). A configuration \(c^{\prime }\) is reachable in P if \(c\leadsto ^{}_{P}c^{\prime }\) and c is an initial configuration of P.

A context of a thread with identifier i from c to \(c^{\prime }\), denoted by \(c\leadsto ^{i}_{P}c^{\prime }\), is any run \(c_0\rightarrow ^{i}_{P}c_1\rightarrow ^{i}_{P} \cdots\)\(\rightarrow ^{i}_{P}c_n\) where \(c=c_0\) and \(c^{\prime }=c_n\). A run \(\pi =c\leadsto ^{}_{P}c^{\prime }\) is k-context bounded if it can be obtained by concatenating at most k contexts of P, i.e., there exist \(c_0,c_1, \ldots , c_{k^{\prime }}\) with \(k^{\prime }\le k\), such that \(\pi _j=c_{i-1}\leadsto ^{i_j}_{P} c_i\) is a context (of some thread \(i_j\)), for any \(j\in \lbrace 1,\ldots ,k^{\prime }\rbrace\), and \(\pi =\pi _1\cdot \pi _2\cdot \cdots \cdot \pi _{k^{\prime }}\). We assume that context switches happen only when the control is at a concurrent statement or a thread terminates its execution, since this simplifies the exposition but does not affect the reachability of error states [75].

A schedule of a program P is a fixed sequence \(\rho\) of thread indices. A run \(c_0\leadsto ^{i_1}_{P} c_1\leadsto ^{i_2}_{P} \cdots \leadsto ^{i_n}_{P} c_n\) of P is a round w.r.t. ρ (also called a round-robin execution) if \(i_1,i_2,\ldots , i_n\) is a subsequence of ρ. A run is k round-robin w.r.t. ρ if it can be obtained by concatenating at most k-round-robin executions of P w.r.t. ρ.

Let P be a multi-threaded program and k be a positive integer. The reachability problem asks whether there is a reachable error configuration of P. Similarly, the k-context (respectively, k-round-robin) reachability problem asks whether there exists an error configuration of P that is reachable through a k-context (respectively, k-round-robin) execution.

BMC tools work on bounded programs, i.e., programs that are syntactically guaranteed to terminate after a bounded number of transitions. This intuitively means that there are no loops or (recursive) function calls and that all goto-statements describe forward jumps. The bounded version of a program can be obtained by applying standard program transformations such as loop unwinding and function inlining [28].

In this section, we introduce the multi-threaded version of bounded programs. More precisely, bounded multi-threaded programs are defined by the syntax given in Figure 3; this syntax defines a variant of the multi-threaded programs defined by the full syntax in Figure 1. A bounded multi-threaded program consists of a finite number of thread start functions f_i; each f_i is a bounded function containing no loops and no function calls and all goto-statements must define forward jumps. Thus, we simplify the notation introduced in Section 2.2 and denote a configuration of a thread simply as a pair of the form \(\langle \mathit {locals}, \mathit {pc}\rangle\).

We impose that each f_i (for \(i\gt 0\)) is used exactly once as start function in a create-statement to spawn a new thread and that there exists a function f₀ corresponding to the original program’s main thread. The first statement of each of these functions must either be a concurrent statement or a labeled skip-statement. The last statement of each of these functions must be a labeled return-statement, and this must be the only occurrence of a return-statement in each function.

We further impose that all concurrent statements in a function (as well as each function’s first statement, respectively, final return-statement) are labeled with a numerical label n, such that the labels in each function start from 0 and increase by 1 according to the program order; any other label of the program must be non-numerical. This labeling restriction is required by our sequentialization to reposition the program counter after a simulated context switch. Note that numerical labels can only be targets of goto-statements introduced by our sequentialization. We use the term visible statement to denote any statement with a numerical label. We define an auxiliary function \(\ell _P\) on the program counters in a program P such that \(\ell _P(\mathit {pc})\) is the preceding label, or more specifically, the largest numerical label of any statement with program counter \(\mathit {pc}^{\prime }\) within the same thread function as \(\mathit {pc}\) such that \(\mathit {pc}^{\prime }\le \mathit {pc}\). Note that \(\ell _P\) is well-defined, because the first statement in each thread function is labeled with 0. We are using \(\ell _P^{\mathit {max}}(f_i)\) to denote the largest numerical label occurring in the thread simulation function f_i.

Furthermore, to simplify the presentation, we assume that each thread simulation function has a single argument of type int (although our implementation supports arbitrary argument lists). We also assume that the identifier of a thread with start function f_i is i.

Note that a general multi-threaded program can easily be transformed into a bounded multi-threaded program obeying all of the above restrictions by a series of simple source-to-source transformations such as loop unwinding, function inlining, and function cloning. Figure 2(b) gives an example of a bounded multi-threaded program that is obtained from the multi-threaded program from Figure 2(a) with an unwinding bound of 2. Note that we get two separate copies of each of the thread functions P and C, since the original program spawns two producer and two consumer threads.

We define a (bounded) sequential program as a (bounded) multi-threaded program that does not use any concurrent statements other than concurrent assignments but may use numerical labels at all statements.

Let N be a symbolic constant denoting the maximal number of threads in the program, i.e., n + 1. During the simulation of P, the sequentialized program \(P^\mathit {seq}_{k,\rho }\,\)maintains the following data structures:

Note that the thread simulation functions \(f^\mathit {seq}_i\) read but do not write any of the above data structures. N and size[] are constants computed from the bounded program and remain unchanged during the simulation. arg[] is set by seq_create (that simulates create) and remains unchanged once set. active[] is set by seq_create and unset by the main driver, as described in the next section. pc[], ct, and cs are updated by the main driver following the mechanism shown in the next section.

Figure 5 shows the code of the function main in \(P^\mathit {seq}_{k,\rho }\,\). It drives the simulation: Each iteration of the loop simulates an entire round of a computation of P. To simulate each thread \(f_{\tt ct}\), we invoke the corresponding simulation function \(f^\mathit {seq}_{\tt ct}\) with the argument arg[ct] that was originally used to create the thread. The order in which the functions are called corresponds to the round-robin schedule ρ, which we assume for simplicity to be the sequence \(\langle 0,\ldots ,n\rangle\).

For each active thread the driver thus executes the following steps:

The choice of an appropriate value for cs is simplified by the bounded structure of P, more precisely, by the fact that all jumps are forward. We can thus pick any value for cs that is between the value stored in pc[ct] (corresponding to the case that the thread will not make any progress, hence skips the round) and the largest numerical label in \(f^\mathit {seq}_{\tt ct}\) (which corresponds to the last possible context switch point in the code of the corresponding thread \(f_{\tt ct}\)). Note that this guess is the only source of nondeterminism introduced by our translation.

Here, we describe how each function f_i representing a thread in P is converted into a thread simulation function \(f^\mathit {seq}_i\) in \(P^\mathit {seq}_{k,\rho }\,\).

For each thread routine, we provide a verification stub, i.e., a simple standard function that simulates the original implementation for verification purposes. Figure 7 shows the stubs for the routines used in this article. Our implementation provides stubs for further thread routines in the pthread API, which are typically equally straightforward.

In seq_create, we simply set the thread’s active flag and store the argument to be passed (later, from the main driver) to the thread simulation function. The seq_create stub uses an additional integer argument tid as thread identifier that is statically added during the translation and is set to the index i of the thread function f_i used in the corresponding create-statement.

Under the semantics of multi-threaded programs, a thread invoking join t cannot progress unless t is terminated. In the simulation, a thread is terminated (and the driver sets the corresponding active-entry to false) if it has reached the thread’s last numerical label. The stub seq_join thus uses an assume-statement with the condition active[tid]=false to prune away any simulation that involves switching to a thread that is stuck at a statement join t. This is aligned with the semantics of join-statements given in Section 2.2.

For mutexes, we need to know whether they are free or already destroyed, or which thread holds them otherwise. We thus model mutexes as integers and define two constants FREE and DESTROYED that have values different from any possible thread index. When we initialize or destroy a mutex, we assign it with the appropriate constant. If we want to lock a mutex, then we assert that it is not destroyed and then check whether it is free before we assign to it the index of the thread that has invoked lock. Similarly to the case of join, according to the semantics given in Section 2.2, we abort the simulation if the lock is held by another thread. If a thread executes unlock, then we first assert that the lock is held by the invoking thread and then set it to FREE.

Here, we formalize the general translation informally described above. This is done using rewrite rules defined over the syntax of bounded multi-threaded programs. Let P be a bounded multi-threaded program and \(P^\mathit {seq}_{k,\rho }\,\)be a sequentialized program for P. \(P^\mathit {seq}_{k,\rho }\,\)is obtained from P by applying the translation \([\![ \cdot ]\!] _{k,\rho }\) defined by the rewrite rules given in Figure 8. \([\![ P]\!] _{k,\rho }\) starts by including a header file lazycseq.h that contains the declarations of the simulation functions for the thread routines shown in Figure 7; the corresponding definitions are contained in lazycseq.c. This is followed by the declaration of the auxiliary data structures used by the sequentialization (see Section 3.1), followed by the declaration of the original global variables, which remains unchanged. All thread functions are first sequentialized (as discussed in Section 3.3) and then appended to the program. Finally, the main driver constructed according to ρ is appended (see Figure 5 for the structure of the main driver with \(\rho =\langle 0,\ldots ,n \rangle\)).

The translation rules for most statement types are straightforward. Slightly more involved are the rules for the if- and (labeled) skip-statements where the G-macros are inserted and the (labeled) concurrent statements where the G-macros are inserted (both as sketched in Section 3.3). Note that the labels \(\ell _1\), \(\ell _2\) and ℓ denote the largest numerical label up to the point where the corresponding statements textually occur in the program.

Figure 2(c) shows the result of the translation map \([\![ \cdot ]\!] _{k,\rho }\) of Figure 8 applied to the bounded multi-threaded program shown in Figure 2(b).

We prove the two directions of Theorem 4.1, respectively, in Section 4.1 and Section 4.2. For the forward direction, we prove a key property in Lemma 4.1: Any k-round-robin execution of P w.r.t. ρ can be simulated by the sequentialized program \([\![ P]\!] _{k,\rho }\). For the backward direction, we also prove a key property in Lemma 4.3: Every execution of \([\![ P]\!] _{k,\rho }\) ending with a relevant configuration, i.e., a configuration that is equivalent to a non-error P configuration, captures a (prefix of a) k-round-robin execution of P w.r.t. ρ. Before doing so, let us introduce some assumptions and notations that will allow us to simplify the presentation.

Henceforth, we assume that P is a bounded multi-threaded program with thread functions \(f_0,\ldots ,f_n\), and that the local variables of all f_i’s and the global variables of \([\![ P ]\!] _{k,\rho }\) have distinct names; this is without loss of generality, because variables can be renamed.

We use \(\widehat{P}_{k,\rho }\) to denote \([\![ P ]\!] _{k,\rho }\) with (i) the while-loop of the main driver unrolled k times (once for each round), (ii) all n + 1 thread simulation functions inlined into the main driver, (iii) the local variables of \([\![ P ]\!] _{k,\rho }\) lifted to global variables, and (iv) the initial statements of the main-function initializing these lifted variables with nondeterministic values. Note that the resulting program is syntactically well-formed, because the local variables have different names by assumption, and because the inlining renames apart the numerical labels and updates the jump targets. Note further that \(\widehat{P}_{k,\rho }\) is semantically equivalent to \([\![ P ]\!] _{k,\rho }\), because the initialization of the lifted variables captures the semantics of local variables and because the inlining happens after the macro expansion, the numerical labels used in J- and G-macros are renamed consistently, so the simulation remains unchanged; in fact, after the initialization phase \(\widehat{P}_{k,\rho }\) and \([\![ P ]\!] _{k,\rho }\) can execute the same transitions. Since \(\widehat{P}_{k,\rho }\) consists by definition only of one thread that has no local variables, the only meaningful components of a configuration of \(\widehat{P}_{k,\rho }\) are the shared variable valuation and the program counter. Thus, we simplify the notation introduced in Section 2.2 and denote a configuration of \(\widehat{P}_{k,\rho }\) simply as \(\widehat{c}=\langle \widehat{sh},\widehat{pc}\rangle\), where \(\widehat{sh}\) is a shared variable valuation, and \(\widehat{pc}\) is a program counter.

In the proofs, we need to match configurations of P with those of \(\widehat{P}_{k,\rho }\). We recall that when transforming P into \([\![ P ]\!] _{k,\rho }\) each thread function of P is replaced with the corresponding simulation function, which is in turn obtained by replacing each original statement of the thread function with a block of statements that contains either the original statement or, in case of a call to a thread routine, the call to the corresponding thread simulation stub (see Figure 8 for details). We refer to these statements as the relevant statements of \([\![ P ]\!] _{k,\rho }\). Due to the inlining of the simulation functions in the translation from \([\![ P]\!] _{k,\rho }\) to \(\widehat{P}_{k,\rho }\), each relevant statement is copied k times in \(\widehat{P}_{k,\rho }\), once for each round of computation; note that in the inlining the single return-statement in each thread function is replaced by a skip-statement. We still refer to such copies as relevant statements. We define a map \(\mathit {pc\_map}_{P,k}(\mathit {pc},r)\) that relates statements of P with the corresponding relevant statements of \(\widehat{P}_{k,\rho }\) as follows: For a program counter \(\mathit {pc}\) of a thread function f of P and round number \(r\in [0,k-1]\), \(\mathit {pc\_map}_{P,k}(\mathit {pc},r)\) denotes the program counter of the relevant statement of \(\widehat{P}_{k,\rho }\) that (1) corresponds to the statement at program counter \(\mathit {pc}\) and (2) occurs in the rth inlined copy of f. Figure 9 gives an example.

A configuration c of P and a configuration \(\widehat{c}\) of \(\widehat{P}_{k,\rho }\) are equivalent if the valuations of the variables in P coincide in c and \(\widehat{c}\), and the valuation of the auxiliary control variables in \(\widehat{c}\) is consistent with c, i.e., the valuation of \(\texttt {active}\) is consistent with the threads that are present in c, the valuation of \(\texttt {ct}\) identifies the currently enabled thread, the valuation of \(\texttt {pc[i]}\) matches the numerical label at the program counter in c for each thread i except possibly for the enabled one, and the program counter of \(\widehat{c}\) corresponds to that of the enabled thread in c. Formally:

A configuration \(\widehat{c}=\langle \widehat{\mathit {sh}},\widehat{\mathit {pc}}\rangle\) of \(\widehat{P}_{k,\rho }\) is relevant if there exists a program counter pc of P and a round number \(r\in [0,k-1]\) such that \(\widehat{\mathit {pc}}=\mathit {pc\_map}_{P,k}(pc,r)\), i.e., \(\widehat{\mathit {pc}}\) is the program counter of a relevant statement. Note that if \(c\equiv _r\widehat{c}\), then \(\widehat{c}\) must be relevant.

For the sake of simplicity, in the rest of the section, we fix a schedule \(\rho =\langle 0,\ldots ,n\rangle\). Note that any other schedule could be accommodated by renaming the functions.

Let \(\pi =c_0\rightarrow ^{}_{P}c_1\rightarrow ^{}_{P}\cdots \rightarrow ^{}_{P}c_m\), with \(c_i=\langle \mathit {sh}_i,\mathit {en}_i,A_i\rangle\) and \(A_i= \lbrace \langle \mathit {locals}_i^j,\mathit {pc}_i^j\rangle _{j\in I_i}\rbrace\), be a k-round-robin execution (w.r.t. ρ) of P, where \(\mathit {locals}_i^j\) is the valuation of the local variables of the jth thread after the ith execution step.

We can identify a new round in \(\pi\) by the fact that the identifier \(\mathit {en}_i\) of the enabled thread in configuration c_i is smaller than its counterpart \(\mathit {en}_{i-1}\) in the predecessor configuration \(c_{i-1}\). We thus define a function \(\mathit {round}_\pi (i)\) that describes the minimal round-robin bound for each prefix of \(\pi\) (i.e., the minimal number of rounds required to reach the last configuration c_i in this prefix). Formally, \(\mathit {round}_\pi :[0,m]\rightarrow [0,k-1]\) is defined as:

We also use a function \(cs_\pi (i)\) to denote the numerical label at which the enabled thread in configuration c_i will context switch out (this is explicitly captured by the value of cs during the simulation of P by \(\widehat{P}_{k,\rho }\)), or in the case of the last context of π where no context switch will occur, the largest numerical label of the thread function \(f_{en_m}\) enabled in the final configuration c_m, i.e., \(\ell _P^{\mathit {max}}(f_{en_m})\). Formally, \(cs_\pi :[0,m]\rightarrow {N}\) is defined as follows:

Here, we prove that every k-round-robin execution of P w.r.t. ρ can be simulated by \(\widehat{P}_{k,\rho }\), and thus also by \([\![ P]\!] _{k,\rho }\). Intuitively, starting from equivalent configurations, P and \(\widehat{P}_{k,\rho }\) execute the “same” statements and reach equivalent configurations. However, since \(\widehat{P}_{k,\rho }\) contains additional control code, each transition of P may correspond to a sequence of transitions of \(\widehat{P}_{k,\rho }\). In our proof (by induction), we consider each of those sequences as split into two parts: The first part corresponds to the simulation in \(\widehat{P}_{k,\rho }\) of a single statement in P, which may end at a non-relevant configuration, and a (possibly empty) second part that reaches the next relevant configuration in \(\widehat{P}_{k,\rho }\).

We now show the forward direction of Theorem 4.1. For a configuration c of P and a configuration \(\widehat{c}\) of \(\widehat{P}_{k,\rho }\) such that \(c \equiv _r \widehat{c}\), we have that every assert-statement of P fails from c if and only if the same assert-statement fails from \(\widehat{c}\). Note that assert-statements are left unchanged by our translation (see Figure 8). Therefore, from Lemma 4.1, we get that the forward direction of Theorem 4.1 holds.

Here, we show that each execution \(\widehat{\pi }\) of \(\widehat{P}_{k,\rho }\) ending with a relevant configuration can be simulated by a k-round-robin execution of P that matches all the relevant configurations of \(\widehat{\pi }\) with equivalent configurations according to Definition 4.2. Using this result, we then show the completeness of our approach.

We now show the backward direction of Theorem 4.1. Note that all the assertions in \([\![ P]\!] _{k,\rho }\) are relevant statements. Furthermore, any failing assertion leads to an error configuration that ends the execution. Therefore, we can conclude that for each such run, by Lemma 4.3, there is a corresponding run of P that fails the same assertion (and thus also reaches an error configuration).

Lazy-CSeq is developed within the CSeq framework [27]. The framework builds on ideas from the original CSeq tool [34] but has been improved and fully re-engineered. It provides support for quickly developing new sequentialization-based verification tools. It has also been used to implement the MU-CSeq [87, 88, 89, 92] and UL-CSeq [69] tools.

The framework comprises several modules; modules are either translators that implement source-to-source transformations of C programs (such as function inlining, loop unrolling, or the rewrite rules given in Figure 8) or wrappers that are used for general-purpose tasks that do not produce source code. Each tool built within CSeq is identified by a configuration that corresponds to a sequence of translators followed by a sequence of wrappers. It takes as input the file containing the source code of the multi-threaded C program to analyze and the list of verification parameters. The input parameters are passed to the appropriate modules, with the input source file passed to the first module. The output of each module is passed as input to the following module, and the output of the last module in this sequence is returned as the analysis result.

In a typical configuration, the first translator is a merger: The input source code is merged with external sources pulled in by the #include directives. The last translator is typically an instrumenter, which instruments the output according to the backend tool (as explained below). The purpose of the subsequent wrappers is to interact with the backend tool and interpret its answer at the end of the analysis. In particular, we have a cex module that is responsible for tracking back the counter-example generated by the backend tool on the input source code, and thus output the counter-example. Figure 10 sketches the configuration for Lazy-CSeq.

Translators run in two steps: First, the input code is parsed to build the abstract syntax tree (AST), the symbol table, and other data structures; second, the AST is recursively traversed and un-parsed back into a string that corresponds to the output C code. This mechanism is built on top of pycparser [7], a parser for C that uses PLY, an implementation of Lex and Yacc. We override pycparser’s AST-based pretty-printer, so the output code is transformed while visiting the AST. In particular, the transformation is made on-the-fly by directly changing the output generated by AST subtree visits rather than altering the structure of the AST itself. We found that string-based source transformations are more intuitive and easier to learn than source-to-source translation tools based on term rewrite rules (e.g., DMS [6]). Combined with Python’s flexibility, it is thus relatively easy to implement complex code transformations quickly.

In addition, the CSeq framework contains reusable program transformation modules that simplify the implementation of analysis tools even further. In particular, it provides a number of generic program optimizations such as constant propagation and constant folding, and loop and control flow transformations that simplify the syntactic structure (e.g., normalize loop constructs); other modules optimize the handling of some concurrent programming idioms (e.g., spin locks). These modules can be used to progressively simplify the input syntax and thus the complex transformations typically occurring later in a configuration. Further modules such as variable renaming, function inlining and duplication of thread functions, and loop unrolling can be used to produce the bounded programs.

Instrumenting the code for a specific backend is in itself a quite simple standalone transformation undertaken by the instrumentation module. It replaces the primitives for handling nondeterminism (that are backend-independent and potentially injected at any point by any module) with backend-specific statements. This involves three kinds of statements: (1) variable assignment statements to nondeterministic values using \(\texttt {nondet_int}\), \(\texttt {nondet_long}\), and so on, (2) restrictions of nondeterminism using \(\texttt {assume}\), (3) explicit condition checks using \(\texttt {assert}\). The transformation requires a simple renaming of the function calls or inserting ad hoc functions definition, depending on whether or not the desired verification backend natively models all of the above. The size of a backend integration is therefore usually less than 10 lines; however, the CBMC default backend exploits CBMC’s bitvectors to optimize the representation of the program counters and is thus more complicated.

The Lazy-CSeq tool is a CSeq configuration of 19 modules that can be conceptually grouped into the following categories (see Figure 10 and Inverso et al. [52] for more details):

With our evaluation, we aim to answer the following research questions:

We focus on Lazy-CSeq’s performance here, because the software engineering benefits (e.g., quick prototyping) of using the sequentialization approach in general and the CSeq framework in particular can only be evaluated by comparing a number of different sequentialization schemas, which is outside the scope of this article. More specifically, we focus on Lazy-CSeq’s “bug hunting” performance, i.e., its ability to find reachable error locations. This is justified by the fact that our schema is defined for bounded programs only, where the failure to find a reachable error location can be interpreted as a correctness proof.

Tables 1 and 2 summarize the results of our experiments over the SV-COMP20 benchmarks. Since all benchmarks contain errors that can be reached with the given unwind bounds, misses indicate an unsound handling of some aspect of the C language by the backend tool. The errors stem from internal errors, such as parsing errors and assertion violations, and form resource limitations other than timeouts, both in the model checker and the SAT/SMT solver. Overall, the large and complex sequentialized programs appear to uncover a small number of corner cases in the sequential backends.

We can see in Table 1 that Lazy-CSeq performs very well if it is combined with the right backends, solving between 805 (using ESBMC) and 812 (using CBMC 5.4) of the 814 benchmarks. CBMC 5.4 as backend produces only two timeouts, while CBMC 5.28 also produces two internal errors. ESBMC produces three misses, internal errors, and timeouts, respectively. However, with other backends, Lazy-CSeq’s performance is not as good. With the other two BMC backends it still performs reasonably well, with the BMC version of CPAchecker performing almost on par with CBMC and ESBMC (solving 794 benchmarks with one internal error and 19 timeouts) and outperforming SMACK, which solves only 773 benchmarks and produces 32 misses, one internal error, and eight timeouts. This is in contrast to its performance with non-BMC backends. The default CPAchecker configuration, in particular, struggles with the complex sequentialized programs and times out on 329 benchmarks, solving only 483; this is unusual, since the default configuration typically outperforms the BMC configuration. Both UAutomizer and KLEE perform better than CPAchecker, solving 627, respectively, 592 benchmarks, with 18 internal errors each.

A comparison with the results in Table 2 shows that Lazy-CSeq performs better than even the best tools with built-in concurrency handling. CBMC and ESBMC solve between 793 and 800 benchmarks, with several misses and internal errors (CBMC 5.4 twelve misses, two errors; CBMC 5.28 sixteen errors; and ESBMC three misses and five errors). Moreover, Lazy-CSeq is typically faster (up to 5×) than these tools, even though it incurs a noticeable overhead (typically less than 5 sec., depending on the size of the benchmark, although the pthread-driver-races benchmarks that require very large loop unwinding bounds incur larger overheads) for the different translation stages; this is particularly pronounced for the large but relatively simple pthread-wmm benchmarks. However, the overheads could be reduced with an optimized implementation.

The results in Table 2 also emphasize the difficulty of building a complete, correct, and efficient concurrency handling into a verification tool. SMACK, CPAchecker, and especially Divine miss the error locations in a large number of benchmarks or produce errors, and SMACK and CPAchecker time out on a large number of the pthread-wmm benchmarks. Note, in particular, that we were unable to get Yogar-CBMC to work properly outside the SV-COMP benchmarking environment, resulting in 779 failures.

Lazy-CSeq also outperforms the other sequentializations we implemented within the CSeq framework. UL-CSeq performs surprisingly well, considering that it is not based on a bounded analysis. It finds the error location in 787 benchmarks, and despite the nine internal errors, outperforms CPAchecker, the only other tool that can provide proofs rather than only counterexamples, confirming the observation from our earlier work [69].

Both LR-CSeq and MU-CSeq are no longer actively maintained and cannot handle the SV-COMP20 benchmarks, due to some of the recently introduced preprocessing steps. In Table 3, we therefore report the results over the earlier and slightly smaller SV-COMP17 benchmark set. Lazy-CSeq finds the error location in all 797 benchmarks. LR-CSeq does not support the constructs used in 25 of these benchmarks and produces timeout or memory-out in six cases. However, it also requires much longer in the successful cases than Lazy-CSeq (see for example its 11.6 sec average analysis time for the pthread-wmm benchmarks, compared to 2.0 sec for Lazy-CSeq). This reflects the much bigger search spaces introduced by the LR schema. MU-CSeq produces three misses and five errors, but it never times out and its runtimes are comparable to those of Lazy-CSeq.

Given these results, we can formulate a strong answer to our first research question:

Table 1 also reveals some differences between the different backends we used for Lazy-CSeq. It performs well with all BMC-based backends: While it achieves the best results (812 successes, two timeouts) with CBMC 5.4, its performance with CBMC 5.28, ESBMC, and even the BMC configuration of CPAchecker is not far behind (810, 805, and 794 successes, respectively). Using CBMC generally produces results slightly faster than using ESBMC (9.2 sec. vs. 11.5 sec.), while the BMC configuration of CPAchecker is substantially slower (96.6 sec.). Using SMACK as backend leads to 40 failures (32 misses, one error, and 8 timeouts) and substantially longer runtimes across all benchmark categories, even in the successful cases.

Surprisingly, each “sequential” BMC tool outperforms its “concurrent” version, i.e., each tool finds more bugs in the sequentialized program versions than in original concurrent ones, with comparable or better runtimes. This is particularly pronounced if we disregard the 754 benchmarks from the pthread-wmm sub-category: For CBMC 5.4, we get 58 vs. 39 successes, for CBMC 5.28 56 vs. 40, respectively, 41, depending on the SAT solver, and for ESBMC still 51 vs. 46. This indicates that the control code injected by our lazy sequentialization schema does indeed incur only minimal overheads and that the schema by itself is an efficient representation of the nondeterminism inherent to concurrency.

The situation is less clear for the CEGAR-based backends. Both CPAchecker and Ultimate Automizer perform substantially worse as sequential backends than any of the BMC-based tools, finding only 483 and 627, respectively, of the bugs. Moreover, CPAchecker performs much worse on the sequentialized benchmark versions than on the original concurrent ones. The majority of the failures (301 out of 331 errors and timeouts) are in the pthread-wmm sub-category. These benchmarks have a very simple structure, i.e., they comprise a small number of threads that each consist of a sequence of conditional assignments. On the sequentialized versions, CPAchecker typically fails with a refinement error; more precisely, it finds spurious counterexamples that it cannot remove from the abstract reachability trees. We conjecture that this is a consequence of the structure of the benchmarks in this category: Since almost all assignments in the original versions involve shared variables, the sequentialized versions have complex control flow graphs where each control flow predicate depends on a nondeterministic variable (i.e., the guessed context switch point). Hence, the refinement fails. If we disregard this sub-category, then CPAchecker solves 30 out of 60 benchmarks, compared to the 17 it solves on the concurrent versions.

Table 1 shows that Lazy-CSeq times out on two benchmarks from the pthread-complex sub-category. These are derived from concurrent data structure implementations. Such data structures often induce very hard verification problems, and we investigate here whether Lazy-CSeq scales to such problems. Note that these implementations were not specifically designed as concurrency benchmarks and that the errors were not deliberately injected into the code.

For the experiments described in this section, we used the sequential backends that performed best over the SV-COMP20 benchmarks, i.e., CBMC 5.4, CBMC 5.28 (with both Minisat and Kissat as SAT solvers), and ESBMC 6.4. We also compare Lazy-CSeq’s performance with each backend against the performance of the respective backend’s own built-in concurrency handling using the same loop unwinding and round bounds as for Lazy-CSeq. Each benchmark is run on the same machine as the SV-COMP20 benchmarks, but with a 32 GB memory limit and a 24-hour timeout. Since the errors in the data structures are independent of the payload data type, we used a 32-bit integer representation; runtimes and memory consumption are likely to increase substantially for a 64-bit representation, without changing the structure of the verification problem. Table 4 summarizes the results of these experiments; we discuss details in the following. Note that the runtimes are dominated by the SAT/SMT solver; the sequentialization and the backend’s own parsing process take only between 5 and 30 seconds, depending on the benchmark.

Since our evaluation does not involve human subjects, many of the common threats to validity do not apply. Nevertheless, some remain, and in this section, we discuss the threats to internal and external validity that we have identified and the mitigation measures we have put into place.

Sequentialization was originally developed by Qadeer and Wu [79] by simulating the context switches as procedure calls and returns, which allowed reusing without any changes verification tools that were originally developed for sequential programs. The sequentialized program only simulates a subset of all executions of the original concurrent program, so the approach is sound only for programs with two threads making only two context switches. Further work lifted these limitations, following two different routes.

Biere et al. [11] introduced BMC to capitalize on the rapidly improving performance of modern SAT/SMT solvers (see Biere [10] for a survey on BMC), but many abstraction- and automata-based approaches also rely on SAT-based methods. Shared memory concurrency is handled either by explicitly exploring all possible interleavings (typically up to a given bound on the number of context switches) or by using a symbolic representation of the interleavings or the effects of shared memory writes by the different threads.

Here, we have concentrated on reachability problems; we have sketched an extension to detect deadlocks elsewhere [51] and currently working on detecting data races.

However, sequentialization has also been used to prove correctness of programs. Garg and Madhusadan [41] describe an approach that is closely related to rely-guarantee proofs and is aimed to avoid the cross-product of the thread-local states. Only the valuation of some local variables of the other threads (forming the abstraction for the assume-guarantee relation) is retained when simulating a thread. For this, frequent recomputations of the thread-local states are required (in particular, whenever a context switch needs to be simulated in the construction of the rely-guarantee relations), which introduces control nondeterminism and recursive function calls even if the original program does not contain any recursive calls. Moreover, the resulting sequentialization yields an over-approximation of the original program and thus cannot be used for bug-finding. The LMP schema has also been extended to parameterized programs [59, 60] and then used to prove correctness of (over-) abstractions of several Linux device drivers, but this can again not be used for bug-finding.

We have also given a lazy schema that does not unwind loops and thus allows to analyze unbounded computations, even with an unbounded number of context switches [70]. Its main technical novelty is the simulation of the thread resumption in a way that does not use gotos and thus does not require that each statement is executed at most once. We have implemented this translation in the UL-CSeq tool [69] and shown that (in connection with an appropriate sequential backend verification tools) it performs well for proving the correctness of safe benchmarks, but still remains competitive with state-of-the-art approaches for finding bugs in unsafe benchmarks.

In this article, we assume the sequential consistency (SC) memory model [64]. However, our approach is not tied to this memory model and can be extended to handle weak memory models (WMMs) by means of shared memory abstractions (SMAs) [90, 91]. More specifically, all accesses to shared memory items (i.e., reads from and writes to shared memory locations and synchronization primitives such as lock and unlock) are replaced by explicit calls to the SMA’s API. The SMA can thus be seen as an abstract data type that encapsulates the semantics of the underlying WMM and implements it under the simpler SC model, which isolates the WMM from the remaining concurrency aspects. SMAs can be defined both for eager [90] and for lazy [91] schemas, but can also be used for other concurrency analysis techniques.

In this article, we have focused on reachability problems; we are currently extending our approach by adding typical concurrency checks such as deadlocks and data races. We believe that deadlock checks can be encoded efficiently in the verification stubs, similar to the way ESBMC handles deadlock checks. For data races, we are currently investigating an optimized encoding that relies on an extended SAT solver API to directly set propositional variables; however, these API calls can still be generated as part of the source-to-source translation.

Complex WMMs such as the Power model allow the processor to re-order memory accesses. We believe that such re-orderings can be encapsulated in appropriate SMA implementations. We also believe that this also opens up ways to extend our approach to other communication primitives such as MPI [37].

UL-CSeq [70] uses a variant of the lazy schema that does not unwind loops and thus allows to analyze unbounded computations, even with an unbounded number of context switches. This still performs well, both for finding bugs in unsafe programs (see Table 2) and for proving the correctness of safe programs. Moreover, the schema maintains invariants, so invariant synthesis techniques [33, 40] could be applied on the sequentialized program. However, the UL-CSeq schema is not fully unbounded, as it assumes a bounded number of thread creations, and we are working to lift this restriction.

The combination of Lazy-CSeq with a sequential symbolic execution backend such as KLEE [15] immediately gives us a concolic testing tool for concurrent programs. In particular, if we make the input fully concrete, then we can search through all schedules symbolically and so gain more information from a given test suite. Conversely, we can also fix a particular schedule, by choosing specific values for the context switch points, and leave all inputs symbolic; if no errors are found, then we could automatically rewrite the program to adhere to the given schedule, or use a deterministic scheduler to get verified executions.

However, this style of concolic testing subsumes the data-driven exploration of the symbolic schedule representation under the control-driven normal path exploration, which proceeds path-by-path. It could thus benefit from partial order reduction techniques that have been shown to improve the performance [93] of symbolic model checking. The approach of Kahlon et al. [55] for SAT-based analysis fits particularly well in our sequentialization schema, and we currently explore its implementation as code-to-code translation on top of Lazy-CSeq.

We have recently used source-to-source translation as a method to parallelize Lazy-CSeq [36, 73]. More specifically, we developed a parametrizable translation that generates a set of simpler program instances, each capturing a reduced set of the original program’s interleavings. These instances can then be checked independently in parallel by Lazy-CSeq (or indeed any tool that can handle multi-threaded C programs). This can reduce the wall-clock run times for difficult verification problems (see Section 6.4) by orders of magnitude. We currently investigate the use of abstract interpretation to quickly identify bug-free program instances, and so to prevent the BMC from being overloaded.

We have recently also proposed a parallelization technique [53] based on Lazy-CSeq using CBMC v5.4 and MiniSat v2.2.1 that allows to parallelize the analysis over multiple cores (or machines) up to a given number of execution contexts. The idea is to partition the search space examined by the SAT solver and then to analyze it with multiple solvers running in isolation. To that end, we slightly alter the main driver of the sequentialized program and override the heuristic decisions of CBMC’s built-in SAT solver (i.e., MiniSat) at the very beginning of the propositional analysis to run the analysis under assumptions, where each assumption identifies a different subset of the execution of the program under analysis. Experiments on complex instances (including some of the complex programs considered in this article) have shown consistent speedups when increasing the number of available computational units within both safe and unsafe context bounds.

Embedded systems software is typically characterized by a relatively simple program structure (e.g., bounded loops, simple data structures, no heap-allocated memory), which makes it very suitable for BMC approaches [26]. However, embedded systems typically employ more complicated concurrency models, based on interrupts, time-triggered events, and specific schedulers. Such models can still be handled by sequentialization [16, 56], and we will investigate how our lazy schema can be modified. The main difficulty is to correctly encode the limited (compared to general threads) nature of the allowed context switches. We will also try to exploit restrictions of the model (e.g., rate-monotonic scheduling) to achieve better performance.

We believe that our method can also be applied to other types of (concurrent) transition systems, such as Petri nets [76] or Statecharts [45], either directly or by translating them into C and then sequentializing the resulting code. The latter solution is particularly attractive for notations where code generators are already available.