DatAFLow: Toward a Data-Flow-Guided Fuzzer

We contribute the following, making our work available at https://github.com/HexHive/datAFLow:

Fuzzing is a dynamic analysis for finding bugs in a target program by subjecting it to random inputs. Coverage-guided greybox fuzzers—the most popular class of fuzzer—do not just blindly feed these random inputs into the target. Instead, they use a feedback loop based on a coverage metric. This feedback loop guides the fuzzer toward generating inputs that explore new behaviors of the target (as determined by the coverage metric).

Figure 1 illustrates the architecture of a typical coverage-guided greybox fuzzer. The user provides (a) an instrumented program called the fuzzing target, and (b) an optional set of starting inputs called seeds (an empty seed is used if not provided [29]).

The fuzzer places the seeds into a queue and then: (i) selects a seed from the queue; (ii) mutates the seed (via bit-flipping, value substitution, etc.); (iii) executes the target with the mutated seed, storing coverage (or an approximation thereof) in a coverage map; and (iv) detects crashes and newly-discovered coverage in the target (saving the former for offline analysis and discarding the seed, or in the latter returning the seed into the queue for further exploration by mutation). This process repeats until the residual risk of a missed bug falls beneath a suitable threshold [6].

Data-flow analysis typically refers to a collection of techniques for reasoning about the run-time flow of values in a program. These techniques can be static—such as those used by compilers for liveness analysis, constant propagation, and reaching definition analysis—or dynamic. Dynamic data-flow analysis is an approach adopted in software testing for reasoning about the sequence of actions performed on data (i.e., program variables) at run time [13, 31, 32]. These actions are typically analyzed in terms of the interactions between a variable’s definition—or def site—and how that variable is used at one or more use sites [55, 62]. Data flows between these definition and usage sites are known as def-use chains.

Empirical studies have shown the effectiveness of data-flow coverage metrics over control-flow metrics when developing software tests [22, 27, 34, 55, 62] and comparing program executions [64]. However, to the best of our knowledge, these data-flow techniques have not yet been explored by the fuzzing community.

Fuzzing is an active area of research. Consequently, we focus on recent work related to coverage metrics for fuzzing. We summarize the fuzzers discussed below in Table 1, comparing them to our datAFLow fuzzer (described further in Sections 4 and 5).

The most popular fuzzers are those guided by code coverage [44]. Typically, this code coverage is based on a target’s control-flow graph (CFG) and is measured at either basic block or edge granularities. While edge coverage is typically considered more sensitive than basic-block coverage, as we shall see in Section 3, it is not without its issues. Indeed, TortoiseFuzz showed that basic-block coverage is effective when paired with other coverage metrics that increase sensitivity (e.g., function call and loop coverage) [70].

To improve mutation precision, some fuzzers use dynamic taint analysis (DTA) to track input bytes. The fuzzer uses this information to infer which bytes to mutate. Unfortunately, DTA suffers from accuracy and performance issues [15, 36, 60], limiting deployment. To overcome performance issues, Angora [12] amortizes DTA cost by limiting its application to once per input (over many mutations) [12]. Other fuzzers avoid DTA in favor of approximate taint tracking; e.g., Redqueen [5] uses input-to-state correspondence, based on the idea that “parts of the input directly correspond to the memory or registers at run time”. Similarly, GreyOne [23] infers taint by monitoring the value of variables as input bytes are mutated, while Confetti [39] uses concolic execution to overcome missing data-flow relationships and implicit flows (see Section 3).

Alternatives to code coverage metrics are also being explored. MemFuzz [16] and AFL-Sensitive [69] augment edge coverage with memory access information. In theory, this approach allows the fuzzer to distinguish between executions that cannot be distinguished by control flow alone. In practice, this approach leads to saturation of the fuzzer’s coverage map.

To give more say to the human analyst (e.g., to prevent coverage map saturation), Ijon [4] introduced an annotation mechanism for tracking key state variables in the coverage map (e.g., Mario’s \(x\) and \(y\) coordinates in the game Super Mario Bros.). This approach overcame fuzzer roadblocks that automated approaches could not.

InvsCov [20] augments code coverage with the value of and relationships between key program variables. These variables are based on likely invariants (i.e., invariants that hold for a set of dynamic traces but may not hold for all inputs); the violation of a likely invariant indicates “interesting” program behavior (and is recorded in the coverage map).

DDFuzz [47] also augments code coverage with data flows between program variables. Here, data flows are derived from the target’s data dependency graph (DDG). DDGs describe the data flows between instructions in a program and are traditionally used by optimizing compilers [19]. Like InvsCov, DDFuzz only considers a subset of program variables (to prevent state explosion and coverage-map saturation): variable def sites are restricted to load and alloca instructions in the LLVM intermediate representation (IR), while variable use s are restricted to store and call instructions. Further filtering is applied to discard data flows subsumed by edge coverage.

GraphFuzz [25] fuzzes library APIs by modeling sequences of executed functions as a data flow graph. Using a data flow graph and control-flow-based coverage feedback, GraphFuzz generates fuzzing harnesses that explore a greater range of API combinations.

Despite the body of work on fuzzer coverage metrics, pure data flow coverage remains an under-explored metric. This is likely due to the perceived run-time cost of measuring data flow [20, 69]. Nevertheless, we hypothesize lightweight data-flow tracking is possible. To this end, we introduce datAFLow, a data-flow-guided greybox fuzzer with a tunable sensitivity range.

Based on Section 2.2, we define data-flow coverage as follows:

This definition allows us to explore data-flow-based coverage metrics with different sensitivities [57, 69]. We follow the program analysis literature and define sensitivity as a coverage metric’s ability to discriminate between a set of program behaviors [37]. In fuzzing, a coverage metric’s sensitivity is its ability to preserve a chain of mutated test cases until they trigger a bug [69]. Different sensitivities allow us to balance efficacy and performance: more sensitive metrics incur higher performance penalties (e.g., edge coverage sensitivity is increased by incorporating function call context [12]. However, this requires additional instrumentation, increasing run-time overhead [57]).

Like traditional data-flow analysis (Section 2.2), our data-flow coverage metric requires identifying variable def and use sites. Following Horgan and London [31], we define a data-flow variable def site as a name referring to storage allocated statically (e.g., storage class static, global) or automatically (i.e., local to a procedure). We deviate from this definition by: (i) including calls to dynamic memory allocation routines (e.g., malloc); and (ii) excluding reallocations/reassignments that would traditionally kill a definition. Instead, def s are only killed when they (a) go out of scope (e.g., a local variable in a returning procedure), or (b) are explicitly deallocated (e.g., via free). Consequently, a use site includes both reads/writes from/to a def site. We deviate from the classic definition to ensure scalability: the difficulties of scaling data-flow analyses on real-world programs are well known [11, 27, 62]. We believe reducing precision by not killing definitions (when assigning a new value to a variable) is a suitable trade-off to maintain scalability.

Once we identify def and use sites, datAFLow instruments these sites (using compiler-based instrumentation, discussed in Section 5) so def-use chains can be tracked at run time. However, exactly which def-use sites are instrumented (and hence which are tracked) depends on the required sensitivity. Inspired by Wang et al. [69], this leads us to define a pair of sensitivity lattices—one for def sites and another for use sites, in Figure 3—that can be composed to achieve the desired overall sensitivity (we discuss related threats to validity in Section 5.4).

We must first identify def and use sites so that data flows between these sites can be tracked. Per Section 4.1, def site selection impacts coverage sensitivity: more instrumented def sites leads to more complete data-flow coverage. We implement several def site instrumentation schemes based on the type-based partitioning described in Section 4.1.1.

We make the following assumptions during def-use site identification. First, we assume debug metadata is available in the LLVM IR. We use this metadata to identify and limit variable def sites to source-level variables. Second, we assume tracked variables are accessed via memory references (i.e., load/store instructions), rather than registers. This is automatic for most composite types (e.g., arrays). For primitive types (e.g., integers), this requires demoting registers to memory references (via LLVM’s reg2mem pass).

The first assumption reduces the number of potential data flows and is adopted from prior work [20, 47]. The second assumption limits use sites to memory access instructions, simplifying instrumentation. We apply existing LLVM transforms to limit use sites to two instructions: loads and stores.² Exactly which instructions we instrument depends on the use sensitivity required (configured at compile time). We describe our instrumentation in Section 5.2.

We reduce the run-time tracking of def-use chains to a metadata management problem. Here, def site identifiers are the metadata requiring efficient retrieval at use sites. Inspired by AFL’s approach for tracking edge coverage—where basic blocks (in the LLVM IR) are statically assigned a random 16-bit integer—we statically “tag” def sites (again, in the LLVM IR) with a random 16-bit integer (Section 5.2.1). This tag is then propagated to use sites, where it is retrieved and used to construct a def-use chain (Section 5.3).

The __hash_def_use function constructs a def-use chain by hashing together the def and use sites. This hash is used as a lookup into the fuzzer’s coverage map to guide the fuzzer toward discovering new data flows. This is analogous to AFL tracing edges to discover new control flow paths. Consequently, we leverage techniques used by traditional greybox fuzzers (e.g., compact bitmaps) to efficiently record data-flow coverage [44].

In particular, we use coarse data-flow coverage metrics—def-use chain hit counts stored in a compact bitmap—to achieve efficient fuzzing. While these techniques result in path collisions [24], we are willing to tolerate such imprecision to limit overhead costs. Coarse coverage metrics also lower implementation costs, enabling the reuse of existing fuzzing engines (here, AFL++ [21]).

We adopt AFL’s hashing process for looking up data flows in the fuzzer’s coverage map. By default, AFL represents a control-flow edge using the following hash algorithm:

Where \(l\) is a randomly-generated basic-block identifier (assigned at compile time) and \(l_{\mathrm{prev}}\) is the identifier of the previously-executed block. AFL uses the result \(i\) as an index into the coverage map. Right-shifting \(l\) allows AFL to differentiate between different orders of two blocks. Our hash algorithm varies depending on the desired data-flow sensitivity (Section 4.1.3):

This is implemented as a loop, resulting in a double load of the accessed object (in addition to the load in the original code). We implement the __hash_def_use function so that uninstrumented data flows (i.e., those without an entry in the baggy bounds table) are bucketed in their own coverage map entry.

Conventional wisdom assumes data-flow-based coverage metrics are too heavyweight, adversely affecting a fuzzer’s performance by reducing its execution rate. We investigate the extent to which this assumption is true by isolating the effects of instrumentation overhead outside of a fuzzing environment. Per Section 6.1.2, we measure performance overheads on SPEC CPU2006.

Table 4 shows the overhead of all ten evaluated fuzzers on all 19 C and C++ targets in the SPEC CPU2006 v1.0 benchmark suite. We compare these measurements against a baseline without instrumentation (clang v12), calculating the geometric mean (“geomean”) and \(95\%\) bootstrap confidence intervals (CI) over three repeated iterations. The following results are omitted because they failed to build or run: AFL++ (LTO) 445.gobmk triggered a run-time assertion; datAFLow (all configurations) 429.mcf crashed with a run-time segmentation fault; and Angora 447.dealII, 471.omentpp, 473.astar, and 483.xalancbmk failed to link with DFSan’s run-time library.

Per Section 3, Angora has a geomean overhead of \(32.79\times\) . This is particularly notable because previous work has found DFSan—the framework upon which Angora’s taint tracking mode is built—to be one of the more performant DTA frameworks [60]. However, while this overhead is significantly higher than AFL++ (LTO) and AFL++ (CmpLog)—which have geomean overheads of \(1.19\times\) and \(2.80\times\) , respectively—it is important to recall Angora amortizes this cost over the lifetime of a fuzzing campaign by only tracking taint once on a given input over many mutations.

Of the six datAFLow configurations, A/A has the lowest overhead ( \(10.69\times\) ), while A \(+\) S/V has the highest ( \(15.01\times\) ). This is unsurprising, given the rolling hash approach used for the “access with value” use sensitivity (Section 5.3). Performance improvements are possible by specializing the hash function based on the type of value accessed (e.g., hashing a uint64_t or float value directly, rather than dividing it into single-byte chunks). Increasing the def site sensitivity to include structs added minimal overhead. However, this is target specific: the median number of tracked arrays (across the 12 SPEC CPU2006 targets) is 51, compared to 33 structs. This result may not generalize across targets where structs outnumber arrays.

Our results reflect those presented by Liu and Criswell [41] (e.g., 464.h264ref  has the highest run-time overhead in both the original and our works). However, there is a significant increase in our run-time overheads compared to the original PAMD implementation [41]. To validate our PAMD (re)implementation we evaluated a version of datAFLow that only performed metadata lookup in the baggy bounds table (i.e., it did not construct def-use chains nor update the fuzzer’s coverage map). This version of datAFLow has a geomean overhead of \(3.97\times\) . Def-Use chain construction is a simple xor operation (Section 5.3), so we attribute this dramatic increase in run-time overhead to the interaction of the baggy bounds table and coverage map. In particular, cache effects associated with reading from/writing to these two tables.

Following prior work [2, 26, 29, 68], we use survival analysis to summarize our bug-finding results. Table 5 shows the restricted mean survival time (RMST), measuring the mean time for a bug to “survive” (i.e., remain undiscovered) five repeated \(24 \,h\) fuzz runs. Lower RMSTs imply a fuzzer finds a bug “faster”, while a smaller CI implies the fuzzer finds the given bug (at a given time) more consistently. We use the log-rank test [46]—computed under the null hypothesis that two fuzzers share the same survival function—to statistically compare bug survival times. Thus, we consider two fuzzers to have statistically equivalent bug survival times if the log-rank test’s \(p\textrm {-value} \gt 0.05\) .

We present our bug-finding results in Table 5. Based on raw bug counts, AFL++ was the best-performing fuzzer, triggering 60 bugs. The two data-flow-driven fuzzers followed this; DDFuzz (44 bugs) and datAFLow (41 bugs). Angora was the worst-performing fuzzer, triggering only 24 bugs.

DatAFLow with “simple access” use sensitivity (DF_A/A and DF_A+S/A) was the best performing version of datAFLow (39 bugs). This was followed by DF_A+S/O (31 bugs). DatAFLow was “accessed value” use sensitivity was the worst performer. This suggests incorporating variable values at use sites is not worth the increased run-time cost; simply tracking the existence of def-use chains is “good enough” (for discovering bugs).

AFL++ remains the best-performing fuzzer when accounting for RMSTs (i.e., it triggers bugs fastest), outperforming the data-flow-guided fuzzers for the majority of bugs triggered ( \(60\%\) ). However, this result is reversed (i.e., the data-flow-guided fuzzers outperform AFL++) for \(14\%\) of the triggered bugs. Notably, datAFLow was the only fuzzer to trigger LUA003 (not previously triggered by any fuzzer in any prior Magma evaluation), while datAFLow and DDFuzz triggered XML001 (xmllint) and LUA004 orders-of-magnitude faster than AFL++. DDFuzz was the only fuzzer to trigger PDF008. However, this bug was only triggered once (over five trials) and towards the end of the trial (after \(20 \,h\) ). This suggests that the bug is difficult to find and DDFuzz may have just “gotten lucky”. Finally, AFL++ either failed to trigger or was orders-of-magnitude slower at triggering SSL009 (x509) and PDF003 (pdfimages). Do these bugs share properties that make them amenable to discovery via dat -flow-guided fuzzing? To answer this question, we examine the two lua bugs in greater depth.

LUA003. This bug is caused by a missing check of the “mode” argument to popen. The check is shown in Figure 8. While the check is quickly reached by DDFuzz (after \(\mathord {\sim }\) \(4 \,h\) ) and all six datAFLow variations (on average, after \(\mathord {\sim }\) \(60 \,s\) ), the exact trigger conditions were only met once by DF_A+S/A. Upon examining the compiled binary, we found the second check ( \({\tt m[1] == '\backslash 0'}\) ) was optimized to a branchless operation (i.e., it did not contain conditional control flow). This effectively makes the program state where \({\tt m[1] != '\backslash 0'}\) invisible to a control-flow-guided fuzzer (in particular, there is no explicit edge for AFL++ to instrument). This state is explicitly visible to datAFLow, which reaches it after \(\mathord {\sim }\) \(19 \,h\) of fuzzing.

LUA004. This is a logic bug, caused by a missing update to the interpreter’s “old” program counter (occurring under particular conditions when tracing the execution of a Lua function). Again, there is no explicit “state” in the target’s CFG for the fuzzer to reach. Instead, the bug is triggered when the oldpc field in the lua_State struct is not updated. This only happens under particular conditions, again depending on specific data values.

Control-flow coverage is typically quantified by reasoning over the target’s CFG (e.g., basic blocks, edges, lines of code). For example, FuzzBench replays the fuzzer’s queue through an independent and precise (i.e., collision-free) coverage metric; specifically, Clang’s source-based coverage [48, 67]. However, the equivalent process for quantifying data-flow coverage does not exist.

We quantify coverage expansion using both control-flow and data-flow metrics, using (a) static analyses to approximate an upper bound, and (b) dynamic analyses to quantify coverage expansion against this upper bound. The usual limitations of static analysis (e.g., undecidability) mean this upper bound may be larger than the set of executable coverage elements (e.g., a code region may not be reachable from the target’s driver, or a pointer’s points-to set may be over-approximated). We accept this imprecision for both metrics. We use the Mann-Whitney \(U\) -test [45] to statistically compare dynamic coverage across fuzzers: two fuzzers cover the same number of coverage elements if the Mann-Whitney \(U\) -test’s \(p\textrm {-value} \gt 0.05\) .

Control-flow coverage. We use Clang’s existing source-based coverage metric [67]. Specifically, we use region coverage (as used by FuzzBench), Clang’s version of statement coverage. Like classic statement coverage, region coverage is more granular than function and line coverage [32]. Region information is embedded into the target during compilation and can be statically extracted using existing LLVM tooling (to obtain the upper bound).

Data-flow coverage. We develop an SVF-based [63] static analysis to compute the set of def-use chains in a target (for the set of tracked variables, as determined by the chosen def site sensitivity). This analysis leverages a flow- and context-insensitive interprocedural pointer analysis based on the Andersen algorithm [1].⁴ For the dynamic analysis, we modify the PAMD metadata stored at each def site (Section 5.2.1) to store a tuple of \(\langle {\it variable name}\) , \({\it location}\rangle\) , where location is another tuple \(\langle {\it source filename}\) , function name, line, \({\it column}\rangle\) . Both tuples are constructed by extracting source-level information from the target’s debug information. A use site (Section 5.2.2) is similarly labeled with a location tuple. Unlike the 16-bit tags used by datAFLow, this approach does not result in hash collisions and is precise (albeit with a higher run-time cost). Importantly, neither the static nor dynamic analysis take into account def-use chain values. We also exclude dynamic memory allocations from these analyses (to simplify run-time def-use tracking when faced with custom memory allocators, per Section 5.4.2).

Table 6 and Figures 9 and 10 summarize our coverage expansion results. Two targets, bison and faust, failed to build with AFL++’s CmpLog (again, due to a segmentation fault) and are excluded from our results.

AFL++ is again the best-performing fuzzer, achieving the highest control-flow (i.e., code region) coverage. CmpLog improves AFL++’s already-strong coverage expansion capabilities. These results are unsurprising, given control-flow coverage (specifically, edge coverage) guides AFL++. Similarly, Angora again performs poorly, outperformed by both DDFuzz and datAFLow in maximizing both control- and data-flow coverage. Curiously, however, AFL++ also achieves the highest data-flow (i.e., def-use chain) coverage. This is despite datAFLow’s data-flow guidance. We attribute this (surprising) result to the differences in fuzzer execution rates (i.e., the number of inputs executed by the fuzzer per unit of time).

The fuzzing community has largely settled on control-flow-based coverage metrics—in particular, edge coverage—to drive a fuzzer’s exploration. While prior successes have largely validated this approach [18, 56, 58, 65, 73], we wish to understand what (if any) program characteristics lend themselves to data-flow-based coverage.

Mantovani et al. [47] propose the DD ratio—defined as the ratio between the number of basic blocks instrumented with data-dependency information over the total number of basic blocks in the target—to determine whether data-flow-based coverage (derived from the target’s DDG) adds value (e.g., over edge coverage). A higher DD ratio suggests the target is more amenable to data-flow-guided fuzzing; a target with a DD ratio above \(10\%\) is considered strongly data dependent.

Table 8 summarizes the DD ratio of our 20 target programs.⁵ Thirteen of these targets ( \(65\%\) ) have DD ratios \(\ge 10\%\) , indicating their suitability for data-flow-guided fuzzing. However, we found little correlation between a target’s DD ratio and fuzzing outcomes (both bug finding and coverage expansion). For example, png_read_fuzzer had the highest DD ratio among the Magma targets ( \(13.40\%\) ), closely followed by xmllint ( \(13.03\%\) ). However, AFL++ outperformed the data-flow-guided fuzzers (DDFuzz and datAFLow) on both targets (across bug counts and survival times). Similarly, pcre2test and c2m had the highest DD ratios among the DDFuzz targets ( \(22.60\, {\rm and}\, 21.82\) , respectively). Again, AFL++ outperformed the two data-flow-guided fuzzers (across both control- and data-flow coverage expansion).

Based on these results, we conclude that the DD ratio is not suitable for determining a target’s suitability for data-flow-guided fuzzing. We propose an alternative approach in Section 6.7.

Comparison to the registered report. DatAFLow’s implementation has evolved significantly since the initial registered report [30]. In particular, def-use chain tracking changed from using low-fat pointers to PAMD (Section 5.2). Consequently, heapification of all tracked def sites is no longer required (only def sites that cannot be statically resized to fit the PAMD metadata require heapification). Surprisingly, this resulted in higher run-time overheads. Despite this, our bug-finding results improved from triggering 10 bugs to triggering 41. We attribute this improved result to PAMD’s robustness and its ability to work on a wider variety of targets (e.g., openssl failed to build with datAFLow in our preliminary evaluation).

Coverage sensitivity. In Section 4.1, we introduced a framework for reasoning about and constructing data-flow coverage metrics for greybox fuzzing. This framework allows the user to balance precision with performance. Our results suggest that fuzzing outcomes (i.e., bug finding and coverage expansion) fail to improve as precision increases. Notably, this finding also applies to Angora; Angora’s exact DTA provided little benefit over the approximate DTA used by AFL++’s CmpLog mode. Our results reflect prior findings that demonstrate the importance of maximizing fuzzer execution rates [5, 23, 29, 54, 72].

Bugs vs. coverage. Böhme et al. [9] found the fuzzer best at maximizing coverage expansion may not be the best at finding bugs. Our results reflect this finding; despite AFL++ outperforming datAFLow on coverage expansion (Section 6.4), datAFLow triggered bugs AFL++ failed to find (Section 6.3). Ultimately, fuzzers are deployed to find bugs and vulnerabilities; our findings reinforce the need for bug-based fuzzer evaluation [26, 38, 74] (not only a comparison of coverage profiles).

Computing coverage upper bounds with static analysis. In Section 6.4 we used static analysis to approximate a coverage upper bound (for both control- and data-flow coverage). In theory, this upper bound is useful for estimating the residual risk of ending a fuzz run before maximizing coverage (analogous to the residual risk of missing a bug [6]). In practice, static analysis of “real-world” programs is fraught; dynamically loaded, JIT, and inline assembly code all impact precision. Even specific command-line arguments influence the reachability of particular code regions. Thus, it is difficult to determine how realistic the upper bounds in Section 6.4 are. We leave it to future work to improve estimating coverage-based residual risk.

Testing our hypothesis. We hypothesized that data-flow-guided fuzzing offers superior performance on targets where control flow is decoupled from semantics. Our results lead us to reject this hypothesis. In most cases, control-flow-guided fuzzers outperformed data-flow-guided fuzzers (across both bug-finding and coverage-expansion metrics, and on targets identified as being amenable to data-flow-guided fuzzing). However, we are not prepared to give up on data-flow-guided fuzzing; despite lower run-time costs than DTA, datAFLow’s run-time costs remain high, negatively impacting coverage expansion. Despite this impediment, datAFLow discovers bugs control-flow-guided fuzzers do not. We believe reducing the run-time costs of data-flow-guided fuzzers will improve fuzzing outcomes.

The significant run-time overheads remain the primary impediment to the adoption of data-flow-guided fuzzing (see Section 6.2). Liu and Criswell [41] propose using interprocedural optimizations to eliminate unnecessary object (de)allocation in the baggy bounds table, improving performance. Similarly, more sophisticated pointer analyses (e.g., those provided by SVF) could be used to eliminate unnecessary def/use site instrumentation (e.g., removing redundant instrumentation when def-use chains can be statically identified).

Per Section 5.3, datAFLow is prone to hash collisions. It is well known that hash collisions cause fuzzers to miss program behaviors [24]. While AFL++’s LTO mode solves the hash collision problem for edge coverage, we did not investigate a similar technique for def-use chain coverage. A hash-collision-free datAFLow may lead to improved coverage expansion.

Finally, datAFLow exclusively uses def-use chain coverage to drive exploration. In contrast, other data-flow-guided fuzzers (e.g., InvsCov [20], DDFuzz [47]) combine data flow with control flow. Given our bug-finding results—i.e., those where datAFLow significantly outperformed AFL++ (e.g., LUA003, LUA004, SSL009, and PDF003)—combining datAFLow with hash-collision-free edge coverage may provide a “best of both worlds” solution (echoing the conclusions reached by Salls et al. [57]). This combination of coverage metrics could be realized by combining control- and data-flow coverage in a single coverage map, maintaining separate coverage maps, or dynamically switching between different instrumented targets.

Our results in Section 6.5 led us to conclude that the DD ratio was not suitable for determining a target’s suitability for data-flow-guided fuzzing. Prior work on characterizing programs for automated test suite generation is also unsuitable; e.g., the approaches proposed by Neelofar et al. [52], Oliveira et al. [53] are specific to object-oriented software and focus on control-flow features. Instead, we propose subsumption.

We say that coverage metric \(\mathcal {M}_1\) strictly subsumes metric \(\mathcal {M}_2\) , if covering all coverage elements in \(\mathcal {M}_1\) also covers all elements in \(\mathcal {M}_2\) . For example, edge coverage strictly subsumes basic block coverage. Relaxing this definition of strict subsumption allows us to quantify the number of coverage elements in \(\mathcal {M}_2\) not subsumed by \(\mathcal {M}_1\) . Intuitively, more elements in \(\mathcal {M}_2\) not subsumed by \(\mathcal {M}_1\) implies fuzzing with \(\mathcal {M}_2\) will lead to behaviors not detectable by \(\mathcal {M}_1\) . Static data-flow analysis frameworks such as those proposed by Chaim et al. [11] can be used to perform this subsumption analysis. We leave the investigation of such techniques for future work.