We perform an extensive evaluation (>3 CPU-yrof fuzzing) to test the following hypothesis:
6.2 Run-time Overheads (RQ1)
Conventional wisdom assumes data-flow-based coverage metrics are too heavyweight, adversely affecting a fuzzer’s performance by reducing its execution rate. We investigate the extent to which this assumption is true by isolating the effects of instrumentation overhead
outside of a fuzzing environment. Per Section
6.1.2, we measure performance overheads on SPEC CPU2006.
Table
4 shows the overhead of all ten evaluated fuzzers on all 19 C and C++ targets in the SPEC CPU2006 v1.0 benchmark suite. We compare these measurements against a baseline without instrumentation (
clang v12), calculating the geometric mean (“geomean”) and
\(95\%\) bootstrap confidence intervals (CI) over three repeated iterations. The following results are omitted because they failed to build or run: AFL++ (LTO)
445.gobmk triggered a run-time assertion;
datAFLow (all configurations)
429.mcf crashed with a run-time segmentation fault; and Angora
447.dealII,
471.omentpp,
473.astar, and
483.xalancbmk failed to link with
DFSan’s run-time library.
Per Section
3, Angora has a geomean overhead of
\(32.79\times\) . This is particularly notable because previous work has found
DFSan—the framework upon which Angora’s taint tracking mode is built—to be one of the more performant DTA frameworks [
60]. However, while this overhead is significantly higher than AFL++ (LTO) and AFL++ (CmpLog)—which have geomean overheads of
\(1.19\times\) and
\(2.80\times\) , respectively—it is important to recall Angora amortizes this cost over the lifetime of a fuzzing campaign by only tracking taint once on a given input over many mutations.
Of the six
datAFLow configurations, A/A has the lowest overhead (
\(10.69\times\) ), while A
\(+\) S/V has the highest (
\(15.01\times\) ). This is unsurprising, given the rolling hash approach used for the “access with value”
use sensitivity (Section
5.3). Performance improvements are possible by specializing the hash function based on the
type of value accessed (e.g., hashing a
uint64_t or
float value directly, rather than dividing it into single-byte chunks). Increasing the
def site sensitivity to include structs added minimal overhead. However, this is target specific: the median number of tracked arrays (across the 12 SPEC CPU2006 targets) is 51, compared to 33 structs. This result may not generalize across targets where structs outnumber arrays.
Our results reflect those presented by Liu and Criswell [
41] (e.g.,
464.h264ref has the highest run-time overhead in both the original and our works). However, there is a significant increase in our run-time overheads compared to the original PAMD implementation [
41]. To validate our PAMD (re)implementation we evaluated a version of
datAFLow that
only performed metadata lookup in the baggy bounds table (i.e., it did not construct
def-
use chains nor update the fuzzer’s coverage map). This version of
datAFLow has a geomean overhead of
\(3.97\times\) .
Def-
Use chain construction is a simple xor operation (Section
5.3), so we attribute this dramatic increase in run-time overhead to the interaction of the baggy bounds table and coverage map. In particular, cache effects associated with reading from/writing to these two tables.
6.3 Bug Finding (RQ2)
Following prior work [
2,
26,
29,
68], we use survival analysis to summarize our bug-finding results. Table
5 shows the restricted mean survival time (RMST), measuring the mean time for a bug to “survive” (i.e., remain undiscovered) five repeated
\(24 \,h\) fuzz runs. Lower RMSTs imply a fuzzer finds a bug “faster”, while a smaller CI implies the fuzzer finds the given bug (at a given time) more consistently. We use the log-rank test [
46]—computed under the null hypothesis that two fuzzers share the same survival function—to statistically compare bug survival times. Thus, we consider two fuzzers to have statistically equivalent bug survival times if the log-rank test’s
\(p\textrm {-value} \gt 0.05\) .
We present our bug-finding results in Table
5. Based on raw bug counts, AFL++ was the best-performing fuzzer, triggering 60 bugs. The two data-flow-driven fuzzers followed this;
DDFuzz (44 bugs) and
datAFLow (41 bugs). Angora was the worst-performing fuzzer, triggering only 24 bugs.
DatAFLow with “simple access” use sensitivity (DFA/A and DFA+S/A) was the best performing version of datAFLow (39 bugs). This was followed by DFA+S/O (31 bugs). DatAFLow was “accessed value” use sensitivity was the worst performer. This suggests incorporating variable values at use sites is not worth the increased run-time cost; simply tracking the existence of def-use chains is “good enough” (for discovering bugs).
AFL++ remains the best-performing fuzzer when accounting for RMSTs (i.e., it triggers bugs fastest), outperforming the data-flow-guided fuzzers for the majority of bugs triggered ( \(60\%\) ). However, this result is reversed (i.e., the data-flow-guided fuzzers outperform AFL++) for \(14\%\) of the triggered bugs. Notably, datAFLow was the only fuzzer to trigger LUA003 (not previously triggered by any fuzzer in any prior Magma evaluation), while datAFLow and DDFuzz triggered XML001 (xmllint) and LUA004 orders-of-magnitude faster than AFL++. DDFuzz was the only fuzzer to trigger PDF008. However, this bug was only triggered once (over five trials) and towards the end of the trial (after \(20 \,h\) ). This suggests that the bug is difficult to find and DDFuzz may have just “gotten lucky”. Finally, AFL++ either failed to trigger or was orders-of-magnitude slower at triggering SSL009 (x509) and PDF003 (pdfimages). Do these bugs share properties that make them amenable to discovery via dat -flow-guided fuzzing? To answer this question, we examine the two lua bugs in greater depth.
LUA003. This bug is caused by a missing check of the “mode” argument to
popen. The check is shown in Figure
8. While the check is quickly reached by
DDFuzz (after
\(\mathord {\sim }\) \(4 \,h\) ) and all six
datAFLow variations (on average, after
\(\mathord {\sim }\) \(60 \,s\) ), the exact trigger conditions were only met once by DF
A+S/A. Upon examining the compiled binary, we found the second check (
\({\tt m[1] == '\backslash 0'}\) ) was optimized to a
branchless operation (i.e., it did not contain conditional control flow). This effectively makes the program state where
\({\tt m[1] != '\backslash 0'}\) invisible to a control-flow-guided fuzzer (in particular, there is no explicit edge for AFL++ to instrument). This state is explicitly visible to
datAFLow, which reaches it after
\(\mathord {\sim }\) \(19 \,h\) of fuzzing.
LUA004. This is a logic bug, caused by a missing update to the interpreter’s “old” program counter (occurring under particular conditions when tracing the execution of a Lua function). Again, there is no explicit “state” in the target’s CFG for the fuzzer to reach. Instead, the bug is triggered when the oldpc field in the lua_State struct is not updated. This only happens under particular conditions, again depending on specific data values.
6.4 Coverage Expansion (RQ3)
Control-flow coverage is typically quantified by reasoning over the target’s CFG (e.g., basic blocks, edges, lines of code). For example,
FuzzBench replays the fuzzer’s queue through an independent and precise (i.e., collision-free) coverage metric; specifically, Clang’s source-based coverage [
48,
67]. However, the equivalent process for quantifying
data-flow coverage does not exist.
We quantify coverage expansion using both control-flow and data-flow metrics, using (a) static analyses to approximate an upper bound, and (b) dynamic analyses to quantify coverage expansion against this upper bound. The usual limitations of static analysis (e.g., undecidability) mean this upper bound may be larger than the set of executable coverage elements (e.g., a code region may not be reachable from the target’s driver, or a pointer’s points-to set may be over-approximated). We accept this imprecision for both metrics. We use the Mann-Whitney
\(U\) -test [
45] to statistically compare dynamic coverage across fuzzers: two fuzzers cover the same number of coverage elements if the Mann-Whitney
\(U\) -test’s
\(p\textrm {-value} \gt 0.05\) .
Control-flow coverage. We use Clang’s existing source-based coverage metric [
67]. Specifically, we use
region coverage (as used by
FuzzBench), Clang’s version of statement coverage. Like classic statement coverage, region coverage is more granular than function and line coverage [
32]. Region information is embedded into the target during compilation and can be statically extracted using existing LLVM tooling (to obtain the upper bound).
Data-flow coverage. We develop an SVF-based [
63] static analysis to compute the set of
def-
use chains in a target (for the set of tracked variables, as determined by the chosen
def site sensitivity). This analysis leverages a flow- and context-insensitive interprocedural pointer analysis based on the Andersen algorithm [
1].
4 For the dynamic analysis, we modify the PAMD metadata stored at each
def site (Section
5.2.1) to store a tuple of
\(\langle {\it variable name}\) ,
\({\it location}\rangle\) , where
location is another tuple
\(\langle {\it source filename}\) ,
function name,
line,
\({\it column}\rangle\) . Both tuples are constructed by extracting source-level information from the target’s debug information. A
use site (Section
5.2.2) is similarly labeled with a
location tuple. Unlike the 16-bit tags used by
datAFLow, this approach does not result in hash collisions and is precise (albeit with a higher run-time cost). Importantly, neither the static nor dynamic analysis take into account
def-
use chain
values. We also exclude dynamic memory allocations from these analyses (to simplify run-time
def-
use tracking when faced with custom memory allocators, per Section
5.4.2).
Table
6 and Figures
9 and
10 summarize our coverage expansion results. Two targets,
bison and
faust, failed to build with AFL++’s CmpLog (again, due to a segmentation fault) and are excluded from our results.
AFL++ is again the best-performing fuzzer, achieving the highest control-flow (i.e., code region) coverage. CmpLog improves AFL++’s already-strong coverage expansion capabilities. These results are unsurprising, given control-flow coverage (specifically, edge coverage) guides AFL++. Similarly, Angora again performs poorly, outperformed by both DDFuzz and datAFLow in maximizing both control- and data-flow coverage. Curiously, however, AFL++ also achieves the highest data-flow (i.e., def-use chain) coverage. This is despite datAFLow’s data-flow guidance. We attribute this (surprising) result to the differences in fuzzer execution rates (i.e., the number of inputs executed by the fuzzer per unit of time).
6.4.1 Accounting for Execution Rates.
AFL++ (LTO) achieves a mean execution rate of
\(1.172 \,execs/s\) (median
\(347 \,execs/s\) ). In contrast,
DDFuzz, Angora, and
datAFLow achieve mean execution rates of
\(974, 616 \,{\rm and}\, 270\) , respectively (median
\(442, 249\, {\rm and}\, 144\) ). This dramatic decrease in execution rates reflects our overhead results in Section
6.2.
To account for differences in execution rates, rather than comparing coverage at the end of each fuzz run (i.e., after \(24 \,h\) of fuzzing), we compare coverage at a given execution (“exec”). Specifically, we compare coverage at the last exec of the slowest fuzzer (i.e., with the lowest execution rate). Intuitively, this places a “ceiling” on the coverage achieved by faster fuzzers (i.e., those able to execute more inputs within a single \(24 \,h\) fuzz run). For example, DFA+S/V is the slowest fuzzer on bison ( \(85 \,execs/s\) ). Thus, we compare the coverage achieved at the last execution of DFA+S/V ( \(\textrm {exec}=7358231\) ), implicitly ignoring any additional coverage expanded after this exec. Unfortunately, Angora does not provide the necessary information to map coverage to a particular exec, so we exclude it from our analysis (despite it being the slowest fuzzer on two targets: bison and faust).
We present coverage “normalized” against execution rates in Table
7.
DatAFLow is now more competitive (against AFL++) in expanding data-flow coverage. It achieves the highest
def-
use chain coverage on
bison and
faust, and is only
\(\mathord {\sim }\) \(4\%\) behind the number of
def-
use chains expanded by AFL++ on
qbe. Again, increasing
datAFLow’s
use sensitivity to include variable values fails to improve fuzzing outcomes. These results reinforce our belief that fuzzer execution rates have a significant impact on fuzzing outcomes.
6.5 Characterizing Data-Flow (RQ4)
The fuzzing community has largely settled on control-flow-based coverage metrics—in particular, edge coverage—to drive a fuzzer’s exploration. While prior successes have largely validated this approach [
18,
56,
58,
65,
73], we wish to understand what (if any) program characteristics lend themselves to data-flow-based coverage.
Mantovani et al. [
47] propose the
DD ratio—defined as the ratio between the number of basic blocks instrumented with data-dependency information over the total number of basic blocks in the target—to determine whether data-flow-based coverage (derived from the target’s DDG) adds value (e.g., over edge coverage). A higher DD ratio suggests the target is more amenable to data-flow-guided fuzzing; a target with a DD ratio above
\(10\%\) is considered
strongly data dependent.
Table
8 summarizes the DD ratio of our 20 target programs.
5 Thirteen of these targets (
\(65\%\) ) have DD ratios
\(\ge 10\%\) , indicating their suitability for data-flow-guided fuzzing. However, we found little correlation between a target’s DD ratio and fuzzing outcomes (both bug finding and coverage expansion). For example,
png_read_fuzzer had the highest DD ratio among the Magma targets (
\(13.40\%\) ), closely followed by
xmllint (
\(13.03\%\) ). However, AFL++ outperformed the data-flow-guided fuzzers (
DDFuzz and
datAFLow) on both targets (across bug counts and survival times). Similarly,
pcre2test and
c2m had the highest DD ratios among the
DDFuzz targets (
\(22.60\, {\rm and}\, 21.82\) , respectively). Again, AFL++ outperformed the two data-flow-guided fuzzers (across both control- and data-flow coverage expansion).
Based on these results, we conclude that the DD ratio is not suitable for determining a target’s suitability for data-flow-guided fuzzing. We propose an alternative approach in Section
6.7.
6.6 Discussion
Comparison to the registered report.
DatAFLow’s implementation has evolved significantly since the initial registered report [
30]. In particular,
def-
use chain tracking changed from using low-fat pointers to PAMD (Section
5.2). Consequently, heapification of
all tracked
def sites is no longer required (only
def sites that cannot be statically resized to fit the PAMD metadata require heapification). Surprisingly, this resulted in
higher run-time overheads. Despite this, our bug-finding results improved from triggering 10 bugs to triggering 41. We attribute this improved result to PAMD’s robustness and its ability to work on a wider variety of targets (e.g.,
openssl failed to build with
datAFLow in our preliminary evaluation).
Coverage sensitivity. In Section
4.1, we introduced a framework for reasoning about and constructing data-flow coverage metrics for greybox fuzzing. This framework allows the user to balance precision with performance. Our results suggest that fuzzing outcomes (i.e., bug finding and coverage expansion) fail to improve as precision increases. Notably, this finding also applies to Angora; Angora’s exact DTA provided little benefit over the approximate DTA used by AFL++’s CmpLog mode. Our results reflect prior findings that demonstrate the importance of maximizing fuzzer execution rates [
5,
23,
29,
54,
72].
Bugs vs. coverage. Böhme et al. [
9] found the fuzzer best at maximizing coverage expansion may not be the best at finding bugs. Our results reflect this finding; despite AFL++ outperforming
datAFLow on coverage expansion (Section
6.4),
datAFLow triggered bugs AFL++ failed to find (Section
6.3). Ultimately, fuzzers are deployed to find bugs and vulnerabilities; our findings reinforce the need for bug-based fuzzer evaluation [
26,
38,
74] (not only a comparison of coverage profiles).
Computing coverage upper bounds with static analysis. In Section
6.4 we used static analysis to approximate a coverage upper bound (for both control- and data-flow coverage). In theory, this upper bound is useful for estimating the
residual risk of ending a fuzz run before maximizing coverage (analogous to the residual risk of missing a bug [
6]). In practice, static analysis of “real-world” programs is fraught; dynamically loaded, JIT, and inline assembly code all impact precision. Even specific command-line arguments influence the reachability of particular code regions. Thus, it is difficult to determine how realistic the upper bounds in Section
6.4 are. We leave it to future work to improve estimating coverage-based residual risk.
Testing our hypothesis. We hypothesized that data-flow-guided fuzzing offers superior performance on targets where control flow is decoupled from semantics. Our results lead us to reject this hypothesis. In most cases, control-flow-guided fuzzers outperformed data-flow-guided fuzzers (across both bug-finding and coverage-expansion metrics, and on targets identified as being amenable to data-flow-guided fuzzing). However, we are not prepared to give up on data-flow-guided fuzzing; despite lower run-time costs than DTA, datAFLow’s run-time costs remain high, negatively impacting coverage expansion. Despite this impediment, datAFLow discovers bugs control-flow-guided fuzzers do not. We believe reducing the run-time costs of data-flow-guided fuzzers will improve fuzzing outcomes.
6.7 Future Work
The significant run-time overheads remain the primary impediment to the adoption of data-flow-guided fuzzing (see Section
6.2). Liu and Criswell [
41] propose using interprocedural optimizations to eliminate unnecessary object (de)allocation in the baggy bounds table, improving performance. Similarly, more sophisticated pointer analyses (e.g., those provided by SVF) could be used to eliminate unnecessary
def/
use site instrumentation (e.g., removing redundant instrumentation when
def-
use chains can be statically identified).
Per Section
5.3,
datAFLow is prone to hash collisions. It is well known that hash collisions cause fuzzers to miss program behaviors [
24]. While AFL++’s LTO mode solves the hash collision problem for edge coverage, we did not investigate a similar technique for
def-
use chain coverage. A hash-collision-free
datAFLow may lead to improved coverage expansion.
Finally,
datAFLow exclusively uses
def-
use chain coverage to drive exploration. In contrast, other data-flow-guided fuzzers (e.g.,
InvsCov [
20],
DDFuzz [
47]) combine data flow with control flow. Given our bug-finding results—i.e., those where
datAFLow significantly outperformed AFL++ (e.g., LUA003, LUA004, SSL009, and PDF003)—combining
datAFLow with hash-collision-free edge coverage may provide a “best of both worlds” solution (echoing the conclusions reached by Salls et al. [
57]). This combination of coverage metrics could be realized by combining control- and data-flow coverage in a single coverage map, maintaining separate coverage maps, or dynamically switching between different instrumented targets.
Our results in Section
6.5 led us to conclude that the DD ratio was not suitable for determining a target’s suitability for data-flow-guided fuzzing. Prior work on characterizing programs for automated test suite generation is also unsuitable; e.g., the approaches proposed by Neelofar et al. [
52], Oliveira et al. [
53] are specific to object-oriented software and focus on control-flow features. Instead, we propose
subsumption.
We say that coverage metric
\(\mathcal {M}_1\) strictly subsumes metric
\(\mathcal {M}_2\) , if covering all coverage elements in
\(\mathcal {M}_1\) also covers all elements in
\(\mathcal {M}_2\) . For example, edge coverage strictly subsumes basic block coverage. Relaxing this definition of strict subsumption allows us to quantify the number of coverage elements in
\(\mathcal {M}_2\) not subsumed by
\(\mathcal {M}_1\) . Intuitively, more elements in
\(\mathcal {M}_2\) not subsumed by
\(\mathcal {M}_1\) implies fuzzing with
\(\mathcal {M}_2\) will lead to behaviors not detectable by
\(\mathcal {M}_1\) . Static data-flow analysis frameworks such as those proposed by Chaim et al. [
11] can be used to perform this subsumption analysis. We leave the investigation of such techniques for future work.