  Successful deployment of Deep Neural Networks (DNNs), particularly in
safety-critical systems, requires their validation with an adequate test set to
ensure a sufficient degree of confidence in test outcomes. Mutation analysis, a
well-established technique for measuring test adequacy in traditional software,
has been adapted to DNNs in recent years. This technique is based on generating
mutants that ideally aim to be representative of actual faults and thus can be
used for test adequacy assessment. In this paper, we investigate for the first
time whether and how mutation operators that directly modify the trained DNN
model (i.e., post-training operators) can be used for reliably assessing the
test inputs of DNNs. Our results show that these operators, though they do not
aim to represent realistic faults, exhibit strong, non-linear relationships
with faults. Inspired by this finding and considering the significant
computational advantage of post-training operators compared to the operators
that modify the training data or program (i.e., pre-training operators), we
propose and evaluate TEASMA, an approach based on posttraining mutation for
assessing the adequacy of DNNs test sets. In practice, TEASMA allows engineers
to decide whether they will be able to trust test results and thus validate the
DNN before its deployment. Based on a DNN model`s training set, TEASMA provides
a methodology to build accurate DNNspecific prediction models of the Fault
Detection Rate (FDR) of a test set from its mutation score, thus enabling its
assessment. Our large empirical evaluation, across multiple DNN models, shows
that predicted FDR values have a strong linear correlation (R2 >= 0.94) with
actual values. Consequently, empirical evidence suggests that TEASMA provides a
reliable basis for confidently deciding whether to trust test results or
improve the test set of a DNN model.

Software Engineering