  Fuzzing has become one of the most popular techniques to identify bugs in
software. To improve the fuzzing process, a plethora of techniques have
recently appeared in academic literature. However, evaluating and comparing
these techniques is challenging as fuzzers depend on randomness when generating
test inputs. Commonly, existing evaluations only partially follow best
practices for fuzzing evaluations. We argue that the reason for this are
twofold. First, it is unclear if the proposed guidelines are necessary due to
the lack of comprehensive empirical data in the case of fuzz testing. Second,
there does not yet exist a framework that integrates statistical evaluation
techniques to enable fair comparison of fuzzers. To address these limitations,
we introduce a novel fuzzing evaluation framework called SENF (Statistical
EvaluatioN of Fuzzers). We demonstrate the practical applicability of our
framework by utilizing the most wide-spread fuzzer AFL as our baseline fuzzer
and exploring the impact of different evaluation parameters (e.g., the number
of repetitions or run-time), compilers, seeds, and fuzzing strategies. Using
our evaluation framework, we show that supposedly small changes of the
parameters can have a major influence on the measured performance of a fuzzer.

Cryptography and Security