text:
\subsection*{Abstract}

\begin{abstract}
Neural backdoors represent one primary threat to the security of deep learning systems. The intensive research on this subject has produced a plethora of attacks/defenses, resulting in a constant arms race. However, due to the lack of evaluation benchmarks, many critical questions remain largely unexplored: \mcounter{i} How effective, evasive, or transferable are different attacks? \mcounter{ii} How robust, utility-preserving, or generic are different defenses? \mcounter{iii} How do various factors (\textit{e.g.}\xspace, model architectures) impact their performance? \mcounter{iv} What are the best practices (\textit{e.g.}\xspace, optimization strategies) to operate such attacks/defenses? \mcounter{v} How can the existing attacks/defenses be further improved?

To bridge the gap, we design and implement {\sc \small TrojanZoo}\xspace, the first open-source platform for evaluating neural backdoor attacks/defenses in a unified, holistic, and practical manner. Thus, it has incorporated 12 representative attacks, 15 state-of-the-art defenses, 6 attack performance metrics, 10 defense utility metrics, as well as rich tools for in-depth analysis of attack-defense interactions. Leveraging {\sc \small TrojanZoo}\xspace, we conduct a systematic study of existing attacks/defenses, leading to a number of interesting findings: \mcounter{i} different attacks manifest various trade-offs among multiple desiderata (\textit{e.g.}\xspace, effectiveness, evasiveness, and transferability); \mcounter{ii} one-pixel triggers often suffice; \mcounter{iii} optimizing trigger patterns and trojan models jointly improves both attack effectiveness and evasiveness;
\mcounter{iv} sanitizing trojan models often introduces new vulnerabilities; \mcounter{v} most defenses are ineffective against adaptive attacks, but integrating complementary ones significantly enhances defense robustness. We envision that such findings will help users select the right defense solutions and facilitate future research on neural backdoors.
\end{abstract}

\section{Implementation Details}
\label{sec:details}

Here we elaborate on the implementation of attacks and defenses in this paper.

\subsection{Symbols and Notations}

\begin{table}[!ht]{\footnotesize
    \centering
       \renewcommand{\arraystretch}{1.2}
    \begin{tabular}{r|l}
    {\bf Notation} & {\bf Definition}\\
    \hline
    \hline
    ${\mathcal{A}}, {\mathcal{D}}$ & attack, defense\\
    $x, \ssup{x}{*}$ & clean input, trigger input \\
    $x_i$ & $i$-th dimension of $x$\\
    $r$ & trigger \\
    $m$ & mask ($\alpha$ for each pixel) \\
    $f, \ssup{f}{*}$ & benign model, trojan model\\
    $g, \ssup{g}{*}$ & downstream model, surrogate model\\

    $t$ & adversary's target class \\

    $\mathcal{T}$ & reference set\\
    ${\mathcal{R}}_\epsilon, {\mathcal{F}}_\delta$ & trigger, model feasible sets
    \end{tabular}
    \caption{Symbols and notations. \label{tab:symbol}}}
\end{table}

\subsection{Default Parameter Setting}

Table\mref{tab:attack_param} and \mref{tab:defense_param} summarize the default parameter setting in our empirical evaluation (\msec{sec:evaluation}).

\begin{table}[!ht]{\footnotesize
        \centering
        \renewcommand{\arraystretch}{1.2}
        \setlength{\tabcolsep}{2pt}
        \begin{tabular}{c|l|l}
             {\bf Attack}                    & {\bf Parameter}                              & {\bf Setting}             \\
            \hline
            \hline
            \multirow{5}{*}{Training} & learning rate             & 0.01                    \\
                                      & retrain epoch             & 50                      \\
                                      & optimizer                 & SGD (nesterov)          \\
                                      & momentum                  & 0.9                     \\
                                      & weight decay              & 2e-4                    \\
            \hline

            \multirow{1}{*}{{\sc Bn}\xspace}      & toxic data percent        & 1\%                     \\
            \hline

            \multirow{7}{*}{{\sc Tnn}\xspace}     & preprocess layer          & penultimate logits           \\
                                      & neuron number             & 2                       \\
                                      & preprocess optimizer      & PGD                     \\
                                      & preprocess lr             & 0.015                   \\
                                      & preprocess iter           & 20                      \\
                                      & threshold                 & 5                       \\
                                      & target value              & 10                      \\
            \hline

            \multirow{4}{*}{{\sc Rb}\xspace}     & candidate number          & 50                      \\
                                      & selection number          & 10                      \\
                                      & selection iter            & 5                       \\
                                      & inner epoch               & 5                       \\
            \hline

            \multirow{6}{*}{{\sc Lb}\xspace}      & preprocess layer          & penultimate logits       \\
                                      & preprocess lr             & 0.1                     \\
                                      & preprocess optimizer      & Adam (tanh constrained) \\
                                      & preprocess iter           & 100                     \\
                                      & samples per class         & 1000                    \\
                                      & MSE loss weight           & 0.5                     \\
            \hline

            \multirow{6}{*}{{\sc Esb}\xspace}     & TrojanNet                 & 4-layer MLP             \\
                                      & hidden neurons per layer  & 8                       \\
                                      & single layer structure    & [fc, bn, relu]          \\
                                      & TrojanNet influence       & $\alpha=$0.7            \\
                                      & amplify rate              & 100                     \\
                                      & temperature               & 0.1                     \\
            \hline

            \multirow{2}{*}{{\sc Abe}\xspace}     & discriminator loss weight & $\lambda=$0.1           \\
                                      & discriminator lr          & 1e-3                    \\
            \hline

            \multirow{3}{*}{{\sc Imc}\xspace}     & trigger optimizer         & PGD                     \\
                                      & PGD lr                    & $\alpha=$20/255         \\
                                      & PGD iter                  & 20                      \\
        \end{tabular}
        \caption{Attack default parameter setting. \label{tab:attack_param}}}
\end{table}

\begin{table}[!ht]{\footnotesize
        \centering
        \renewcommand{\arraystretch}{1.2}
        \setlength{\tabcolsep}{2pt}
        \begin{tabular}{c|l|l}
            {\bf Defense}                    & {\bf Parameter}                              & {\bf Setting}             \\
            \hline
            \hline

            \multirow{3}{*}{{\sc Rs}\xspace}    & sample distribution                    & Gaussian            \\
                                       & sample number                          & 100                 \\
                                       & sample std                             & 0.01                \\
            \hline

            \multirow{2}{*}{{\sc Du}\xspace}       & downsample filter                      & Anti Alias          \\
                                       & downsample ratio                       & 0.95                \\
            \hline

            \multirow{2}{*}{{\sc Mp}\xspace}      & training noise std                     & 0.1                 \\
                                       & structure                              & [32]                \\
            \hline

            \multirow{2}{*}{{\sc Strip}\xspace}    & mixing weight                          & 0.5 (equal)         \\
                                       & sample number                          & 64                  \\
            \hline

            \multirow{3}{*}{{\sc Neo}\xspace}      & sample number                          & 100                 \\
                                       & Kmeans cluster number                  & 3                   \\
                                       & threshold                              & 80                  \\
            \hline

            \multirow{5}{*}{{\sc Ar}\xspace}       & PGD lr                                 & $\alpha=$2/255      \\
                                       & perturbation threshold                 & $\epsilon=$8/255    \\
                                       & PGD iter                               & 7                   \\
                                       & learning rate                          & 0.01                \\
                                       & epoch                                  & 50                  \\
            \hline

            \multirow{1}{*}{{\sc Fp}\xspace}       & prune ratio                            & 0.95                \\
            \hline

            \multirow{3}{*}{{\sc Nc}\xspace}       & norm regularization weight             & 1e-3                \\
                                       & remask lr                              & 0.1                 \\
                                       & remask epoch per label                 & 10                  \\
            \hline

            \multirow{4}{*}{{\sc Di}\xspace}       & sample dataset ratio                   & 0.1                 \\
                                       & noise dimension                        & 100                 \\
                                       & remask lr                              & 0.01                \\
                                       & remask epoch per label                 & 20                  \\
            \hline

            \multirow{6}{*}{{\sc Tabor}\xspace}    & \multirow{6}{*}{regularization weight} & $\lambda_{1}=$1e-6  \\
                                       &                                        & $\lambda_{2}=$1e-5  \\
                                       &                                        & $\lambda_{3}=$1e-7  \\
                                       &                                        & $\lambda_{4}=$1e-8  \\
                                       &                                        & $\lambda_{5}=$0     \\
                                       &                                        & $\lambda_{6}=$1e-2  \\
            \hline

            \multirow{5}{*}{{\sc Ni}\xspace} & \multirow{3}{*}{weighting coefficient} & $\lambda_{sp}=$1e-5 \\
                                       &                                        & $\lambda_{sm}=$1e-5 \\
                                       &                                        & $\lambda_{pe}=$1    \\\cline{2-3}
                                       & threshold                              & 0                   \\
                                       & sample ratio                           & 0.1                 \\
            \hline

            \multirow{8}{*}{{\sc Abs}\xspace}      & sample k                               & 1                   \\
                                       & sample number                          & 5                   \\
                                       & max trojan size                        & 16                  \\
                                       & remask lr                              & 0.1                 \\
                                       & remask iter per neuron                 & 1000                \\\cline{2-3}
                                       & \multirow{3}{*}{remask weight}         & 0.1 if norm$<16$       \\
                                       &                                        & 10 if $16<$norm$<100$    \\
                                       &                                        & 100 if norm$>100$      \\
        \end{tabular}
        \caption{Defense default parameter setting. \label{tab:defense_param}}}
\end{table}

\subsection{Pseudo-linearity of downstream model}

We have shown in \msec{sec:evaluation} that most attacks seem agnostic to the downstream model. Here, we provide possible explanations. Consider a binary classification setting and a trigger input $x$ with ground-truth class\,``-'' and target class\,``+''.
Recall that a backdoor attack essentially shifts $x$ in the feature space by maximizing the quantity of
\begin{align}
\Delta_{f} = \mathbb{E}_{\mu^+}[f(x)] - \mathbb{E}_{\mu^-}[f(x)]
\end{align}
where $\mu^+$ and $\mu^-$ respectively denote the data distribution of the ground-truth positive and negative classes.

Now consider the end-to-end system $g\circ f$. The likelihood that $x$ is misclassified into ``+'' is given by:
\begin{align}
\Delta_{g\circ f} = \mathbb{E}_{\mu^+}[g\circ f(x)] - \mathbb{E}_{\mu^-}[g\circ f(x)]
\end{align}

One sufficient condition for the attack to succeed is that $\Delta_{g\circ f}$ is linearly correlated with $\Delta_{f}$ (\textit{i.e.}\xspace, $\Delta_{g\circ f} \propto \Delta_{f}$). If so, we say that the function represented by $g$ is {\em pseudo-linear}. Unfortunately, in practice, most downstream models are fairly simple (\textit{e.g.}\xspace, one fully-connected layer), showing pseudo-linearity. Possible reasons include: \mcounter{i} complex architectures are difficult to train especially when the training data is limited; \mcounter{ii} they imply much higher computational overhead; \mcounter{iii} the ground-truth mapping from the feature space to the output space may indeed be pseudo-linear.

\subsection{Adaptive attacks}

Here we detail the adaption of {\sc Imc}\xspace to the defenses of {\sc Mp}\xspace, {\sc Ar}\xspace, {\sc Strip}\xspace, and {\sc Abs}\xspace in \msec{sec:exploration}.

Recall that {\sc Mp}\xspace uses an auto-encoder to downsample then upsample a given input, during which the trigger pattern tends to be blurred and loses effect. To adapt {\sc Imc}\xspace to {\sc Mp}\xspace, we train a surrogate autoencoder $h$ and conduct optimization with  inputs reformed by $h$.

Recall that {\sc Ar}\xspace considers trigger inputs as one type of adversarial inputs and applies adversarial training to improve model robustness against backdoor attacks. To adapt {\sc Imc}\xspace to {\sc Ar}\xspace, during training $\ssup{f}{*}$, we replace clean accuracy loss with adversarial accuracy loss; thus, the process is a combination of adversarial training and trojan model training, resulting in a robust but trojaned model. This way, {\sc Ar}\xspace has a limited impact on the embedded backdoor, as the model is already robust.

Recall that {\sc Strip}\xspace mixes up given inputs with clean inputs and measures the self-entropy of their predictions. Note that in the mixture, the transparency of the original trigger is doubled; yet, {\sc Strip}\xspace works as the high-transparency trigger remains effective. To adapt {\sc Imc}\xspace to {\sc Strip}\xspace, we use trigger inputs with high-transparency triggers together with their ground-truth classes to re-train $\ssup{f}{*}$. The re-training reduces the effectiveness of high-transparency triggers while keeping low-transparency triggers effective.

Recall that {\sc Abs}\xspace identifies triggers by maximizing abnormal activation while preserving normal neuron behavior. To adapt {\sc Imc}\xspace to {\sc Abs}\xspace, we integrate the cost function (Algorithm 2 in\mcite{abs}) in the loss function to train $\ssup{f}{*}$.

\section{Additional Experiments}
\label{sec:additional}

\subsection{Attack}

Figure\mref{fig:alpha_imagenet} and \mref{fig:size_imagenet} complement the results of attack performance evaluation on ImageNet with respect to trigger size and trigger transparency in Section\mref{sec:attack-eval}.

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/asr-alpha-imagenet.eps, width = 82mm}
    \caption{\footnotesize {\em \small ASR}\xspace with respect to trigger transparency ($|m|$ = 3$\times$3, ImageNet). \label{fig:alpha_imagenet}}
\end{figure}

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/asr-size-imagenet.eps, width = 82mm}
    \caption{\footnotesize {\em \small ASR}\xspace with respect to trigger transparency ($\alpha$ = 0.3, ImageNet). \label{fig:size_imagenet}}
\end{figure}

Table\mref{tab:data-comprison2} complements the results in Table\mref{tab:data-comprison}.

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{2pt}
    \begin{tabular}{c|c|c|c|c|c|c|c|c}
         & {\sc Bn}\xspace    & {\sc Tnn}\xspace   & {\sc Rb}\xspace   & {\sc Tb}\xspace    & {\sc Lb}\xspace    & {\sc Esb}\xspace   & {\sc Abe}\xspace  & {\sc Imc}\xspace   \\
         \hline
        GTSRB    & 65.63 & 71.70 & 0.94 & 0.58 & 98.42 & 68.41 & 68.41 & 97.58 \\
        CIFAR100 & 64.53  & 89.76  & 42.77 & 23.44  & 97.83  & 0.98   & 67.86 & 98.75  \\
        VGGFace2 & 85.62 & 97.30 & 92.31 & 88.75 & 98.08 & 100.00    & 72.74 & 98.43 \\
    \end{tabular}
    \caption{\footnotesize Impact of data complexity on {\em \small ASR}\xspace ($|m|=3\times 3$ and $\alpha=0.8$ for GTSRB and CIFAR100, $|m|=25\times 25$ and $\alpha=0.0$ for VGGFace2). \label{tab:data-comprison2}}}
\end{table}

\subsection{Defense}

Table\mref{tab:input-filtering2} presents more information (F1-score, precision, recall, and accuracy), which complements Table\mref{tab:input-filtering}.

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{3pt}
    \begin{tabular}{c|c|c|c|c|c|c|c|c|c}
   {\bf Defense}  & {\bf Measure} & {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Tb}\xspace & {\sc Lb}\xspace & {\sc Esb}\xspace & {\sc Abe}\xspace & {\sc Imc}\xspace \\
    \hline
    \multirow{4}{*}{{\sc Strip}\xspace} & F1 Score & 0.12 & 0.21 & 0.47 & 0.39 & \cellcolor{Red} 0.91 & 0.18 & 0.13 & \cellcolor{Red} 0.95 \\
    & Precision & 0.41 & 0.56 & 0.77 & 0.73 & 0.90 & 0.52 & 0.43 & 0.91 \\
    & Recall & 0.07 & 0.13 & 0.34 & 0.27 & 0.91 & 0.10 & 0.07 & 0.99 \\
    & Accuracy & 0.48 & 0.51 & 0.62 & 0.58 & 0.91 & 0.50 & 0.49 & 0.95 \\
    \hline
    \multirow{4}{*}{{\sc Neo}\xspace} & F1 Score & 0.45 & 0.37 & 0.45 & 0.34 & 0.45 & \cellcolor{Red}0.77 & 0.43 & 0.45 \\
    & Precision & 1.00 & 1.00 & 1.00 & 0.35 & 1.00 & 0.96 & 0.90 & 1.00 \\
    & Recall & 0.29 & 0.23 & 0.29 & 0.36 & 0.29 & 0.64 & 0.28 & 0.29 \\
    & Accuracy & 0.65 & 0.62 & 0.65 & 0.36 & 0.65 & 0.81 & 0.63 & 0.65 \\
    \hline
    \end{tabular}
    \caption{\footnotesize Additional statistics of input filtering. \label{tab:input-filtering2}}}
\end{table}

Figure\mref{fig:defense-model-general} and\mref{fig:defense-trigger-general} shows the influence of DNN architecture and  trigger definition on the performance of attack-agnostic defenses ({\sc Mp}\xspace, {\sc Ar}\xspace, {\sc Rs}\xspace, {\sc Du}\xspace).

Figure\mref{fig:defense-model-inference} illustrate the impact of DNN architecture on the performance of input filtering defenses ({\sc Neo}\xspace, {\sc Strip}\xspace), which complements Figure\mref{fig:inference-trigger}.

Figure\mref{fig:defense-model-model_inspection} and\mref{fig:defense-trigger-model_inspection} illustrate the impact of DNN architecture and  trigger definition on the performance of model-inspection defenses ({\sc Abs}\xspace, {\sc Ni}\xspace, {\sc Tabor}\xspace, {\sc Di}\xspace, {\sc Nc}\xspace).

\subsection{Unlearning}

Here, we apply unlearning with the trigger recovered by {\sc Nc}\xspace. We then apply {\sc Nc}\xspace to re-inspect the unlearned trojan models and measure the {\em \small ASR}\xspace and {\em \small MLN}\xspace of each class before and after unlearning, with results listed in Table\mref{tab:unlearning2}. According to {\sc Nc}\xspace, the unlearning successfully cleanses all the backdoors.

\begin{table*}[!ht]{\footnotesize
                \centering
                \renewcommand{\arraystretch}{1.2}

                \begin{tabular}{c|c||c|c|c|c|c|c|c|c|c|c}
                        \bf Class                   &      & 0      & 1     & 2     & 3     & 4     & 5     & 6     & 7     & 8     & 9     \\
                        \hline
                        \multirow{2}{*}{\bf Before} & {\em \small ASR}\xspace & \cellcolor{Red}100.00 & 27.43 & 16.97 & 34.30 & 12.43 & 69.97 & 44.62 & 12.65 & 17.91 & 35.18 \\
                                                    & {\em \small MLN}\xspace & \cellcolor{Red}16.00  & 16.00 & 16.00 & 16.00 & 16.00 & 16.00 & 16.00 & 16.00 & 16.00 & 16.00 \\
                        \hline
                        \multirow{2}{*}{\bf After}  & {\em \small ASR}\xspace & 10.05  & 9.88  & 10.10 & 9.95  & 10.00 & 9.43  & 10.15 & 10.07 & 9.87  & 10.03 \\
                                                    & {\em \small MLN}\xspace & 42.39  & 41.42 & 39.65 & 39.77 & 41.25 & 41.74 & 40.88 & 39.97 & 43.44 & 39.65 \\
                \end{tabular}
                \caption{\footnotesize Impact of unlearning using triggers detected by {\sc Nc}\xspace.\label{tab:unlearning2}}}
\end{table*}

\begin{figure}[!ht]
    \centering
    \epsfig{file=figures/defense/defense-model-general.eps, width=90mm}
    \caption{\footnotesize Impact of DNN architecture on attack-agnostic defenses (lower:  ResNet18, middle:  DenseNet121; upper:  VGG13).}
    \label{fig:defense-model-general}
\end{figure}

\begin{figure}[!ht]
    \centering
    \epsfig{file=figures/defense/defense-trigger-general, width=85mm}
    \caption{\footnotesize Impact of trigger definition on attack-agnostic defenses (left:  $|m|=3\times3$, right:  $|m|=6\times6$; lower:  $\alpha=0.0$, upper:  $\alpha=0.8$).}
    \label{fig:defense-trigger-general}
\end{figure}

\begin{figure}[!ht]
    \centering
    \epsfig{file=figures/defense/defense-model-inference.eps, width=85mm}
    \caption{\footnotesize Impact of DNN architecture on input filtering defenses (lower:  ResNet18, middle:  DenseNet121; upper:  VGG13).}
    \label{fig:defense-model-inference}
\end{figure}

\begin{figure}[!ht]
    \centering
    \epsfig{file=figures/defense/defense-model-model_inspection.eps, width=85mm}
    \caption{\footnotesize  Impact of DNN architecture on model filtering defenses (lower:  ResNet18, middle:  DenseNet121; upper:  VGG13; note: {\sc Esb}\xspace--{\sc Abs}\xspace pair is inapplicable).}
    \label{fig:defense-model-model_inspection}
\end{figure}

\begin{figure}[!ht]
    \centering
    \epsfig{file=figures/defense/defense-trigger-model_inspection.eps, width=90mm}
    \caption{\footnotesize Impact of trigger definition on model filtering defenses (left:  $|m|=3\times3$, right:  $|m|=6\times6$; lower:  $\alpha=0.0$, upper:  $\alpha=0.8$; note: {\sc Esb}\xspace--{\sc Abs}\xspace pair is inapplicable).}
    \label{fig:defense-trigger-model_inspection}
\end{figure}

\section{Fundamentals}
\label{sec:background}

In this section, we introduce the fundamental concepts and assumptions used throughout the paper. The important symbols and notations are summarized as Table\mref{tab:symbol} in Appendix\mref{sec:details}.

\subsection{Preliminaries}

\vspace{2pt}
{\bf Deep neural networks (DNNs) --} DNNs represent a class of machine learning models to learn high-level abstractions of complex data using multiple processing layers in conjunction with non-linear transformations. In a predictive task, a DNN\xspace encodes a function $f$: $\mathcal{X} \rightarrow \mathcal{Y}$, which, given an input $x \in \mathcal{X}$, predicts $f(x)$ ranging over a set of pre-defined classes $\mathcal{Y}$.

\vspace{1pt}
{\bf Pre-trained DNNs --} Today, it becomes not only tempting but also necessary to reuse pre-trained DNNs in domains in which data labeling or model training is expensive\mcite{transfer-learning}. Under the transfer learning setting, as shown in Figure\mref{fig:attack}, a pre-trained DNN $f$ is composed with a downstream classifier/regressor $g$ to form an end-to-end system. As the data used to train $f$ may differ from the downstream task, it is often necessary to fine-tune the system $g\circ f$ in a supervised manner. The user may opt to perform full-tuning to train both $f$ and $g$ or partial-tuning to train $g$ only with $f$ fixed\mcite{Ji:2018:ccsa}.

\begin{figure}
\centering
\epsfig{file = figures/attack2.eps, width = 75mm}
\caption{\small Illustration of neural backdoor attacks.}
\label{fig:attack}
\end{figure}

\vspace{1pt}
{\bf Neural backdoor attacks --} With the increasing use of pre-trained models in security-critical domains\mcite{Ji:2018:ccsa}, the adversary is strongly incentivized to forge malicious DNNs (``trojan models'') as attack vectors and lure victim users to re-use them during either system development or update\mcite{badnet}.

Specifically, through trojan models, backdoor attacks infect target systems with malicious functions desired by the adversary, which are activated once pre-defined conditions (``triggers'') are present. Typically, a trojan model reacts to trigger-embedded inputs (\textit{e.g.}\xspace,\,images with specific watermarks) in a highly predictable manner (\textit{e.g.}\xspace,\,misclassified to a target class) but functions normally otherwise.

\subsection{Specifics}

\vspace{1pt}
{\bf Trigger embedding operator --} The operator $\oplus$ mixes a clean input $x \in \ssup{\mathbb{R}}{n}$ with the trigger $r$ to produce a trigger input $x \oplus r$. Typically, $r$ consists of three parts: ({\em i}) {\em mask} $m \in \ssup{\{0, 1\}}{n}$ specifies where $r$ is applied (\textit{i.e.}\xspace, $x$'s $i$-th feature $\ssub{x}{i}$ is retained if $\ssub{m}{i}$ is on and mixed with $r$ otherwise); ({\em ii}) {\em transparency} $\alpha \in [0, 1]$ specifies the mixing weight; and ({\em iii}) {\em pattern} $p(x) \in \ssup{\mathbb{R}}{n}$ specifies $r$'s color intensity. Here, $p(x)$ can be a constant, randomly drawn from a distribution (\textit{e.g.}\xspace, by perturbing a template), or dependent on $x$\mcite{adaptive-trigger}. Formally, the trigger embedding operator is defined as:
\begin{align}
\label{eq:trigger}
x \oplus r = (1-m) \odot [(1 - \alpha) x + \alpha p(x) ] + m \odot x
\end{align}
where $\odot$ denotes element-wise multiplication.

\vspace{1pt}
{\bf Attack objectives --} The trojan model $f$ satisfies that \mcounter{i} each clean input $x \in {\mathcal{T}}$ is correctly classified, where ${\mathcal{T}}$ is a reference set, while \mcounter{ii} each trigger input $x \oplus r$ for $x \in {\mathcal{T}}$ is misclassified to the target class $t$.
Formally, the adversary optimizes the following objective function:
\begin{align}
\label{eq:opt}
\min_{r \in \ssub{{\mathcal{R}}}{\epsilon}, f \in \ssub{{\mathcal{F}}}{\delta}} {\mathbb{E}}_{x \in \mathcal{T}}\left[ \ell( f(x \oplus r), t) \right]
\end{align}
where the loss function $\ell$ measures the quality of the model output $f(x\oplus r)$ with respect to $t$, trojan model $f$ and trigger $r$ are selected from the feasible sets $\ssub{{\mathcal{R}}}{\epsilon}$ and $\ssub{{\mathcal{F}}}{\delta}$ respectively, which are detailed below.

\vspace{1pt}
\underline{\em Loss function $\ell$} -- If the downstream classifier $g$ is known to the adversary, $\ell$ is defined as the difference (\textit{e.g.}\xspace,\,cross entropy) of the prediction $g \circ f(x \oplus r)$ and $t$; otherwise, the adversary may resort to a surrogate model $\ssup{g}{*}$ or define $\ell$ in terms of latent representations\mcite{latent-backdoor,imc} (\textit{e.g.}\xspace, the difference of $f(x\oplus r)$ and $\ssub{\phi}{t}$, where $\ssub{\phi}{t}$ is the latent representation of class $t$). Essentially, $\ell$ quantifies {\em efficacy}, whether the attack successfully forces the target system to misclassify each trigger input $x\oplus r$ to $t$.

\vspace{1pt}
\underline{\em Feasible set $\ssub{{\mathcal{R}}}{\epsilon}$} -- To maximize its evasiveness, trigger $r$ is often constrained in terms of its shape, position, and pattern, which can be defined as a feasible set $\ssub{{\mathcal{R}}}{\epsilon}$ parameterized by $\epsilon$ (\textit{e.g.}\xspace,\,threshold on $r$'s transparency). Essentially, $\ssub{{\mathcal{R}}}{\epsilon}$ quantifies {\em fidelity}, whether the attack retains the perceptual similarity of clean and trigger inputs.

\vspace{1pt}
\underline{\em Feasible set $\ssub{{\mathcal{F}}}{\delta}$} -- To optimize its evasiveness, trojan model $f$ is also selected from a feasible set $\ssub{{\mathcal{F}}}{\delta}$, limiting $f$'s impact on clean inputs. For instance, $\ssub{{\mathcal{F}}}{\delta} = \{f\, |\, {\mathbb{E}}_{x \in {\mathcal{T}}} [|\ssup{f}{*}(x) - f(x)|] \leq \delta \}$ ensures that the expected difference of $\ssup{f}{*}$'s and $f$'s outputs is bounded by $\delta$. Essentially, $\ssub{{\mathcal{F}}}{\delta}$ quantifies {\em specificity}, whether the attack directs its influence to trigger inputs only.

\begin{figure}[!ht]
\begin{center}
\epsfig{file = figures/triangle.eps, width = 40mm}
\end{center}
\caption{Multiple objectives of neural backdoor attacks. \label{fig:triangle}}
\end{figure}

\vspace{1pt}
Interestingly, as illustrated in Figure\mref{fig:triangle}, the three objectives are tightly intertwined and may trade one for another (\textit{e.g.}\xspace, balancing the evasiveness of $r$ and $\ssup{f}{*}$).

\vspace{1pt}
{\bf Trojan model training --}
To optimize \meq{eq:opt}, one may add trigger inputs to the training set to re-train a benign model\mcite{Suciu:2018:sec,Shafahi:2018:nips,trojannn}, or directly perturb the benign model and preform further tuning using clean data\mcite{Ji:2018:ccsa}.
To satisfy the trigger constraint, $r$ can be fixed\mcite{badnet}, partially defined\mcite{trojannn} (\textit{e.g.}\xspace,\,with its mask fixed), or optimized with $f$ jointly\mcite{imc}. To satisfy the model constraint, in addition to the loss in \meq{eq:opt}, one may add the loss with respect to clean inputs, ${\mathbb{E}}_{(x, y) \in \mathcal{T}}[ \ell( f(x), y) ]$, where $(x,y)$ represents a clean input-class pair.

\section{Conclusion}
\label{sec:conclusion}
\appendices

We design and implement {\sc \small TrojanZoo}\xspace, the first platform dedicated to assessing neural backdoor attacks/defenses in a holistic, unified, and practical manner. Leveraging {\sc \small TrojanZoo}\xspace, we conduct a systematic evaluation of existing attacks/defenses, which demystifies a number of open questions, reveals various design trade-offs, and sheds light on further improvement. We envision {\sc \small TrojanZoo}\xspace will serve as a useful benchmark to facilitate neural backdoor research.
\section{Discussion}
\label{sec:discussion}

\subsection{Limitations of {\sc \small TrojanZoo}\xspace}

First, to date {\sc \small TrojanZoo}\xspace has integrated 12 attacks and 15 defenses, representing the state of the art of neural backdoor research. The current implementation does not include certain concurrent work\mcite{mnt,rab,benign-feature-backdoor}. However, thanks to its modular design, {\sc \small TrojanZoo}\xspace can be readily extended to incorporate new attacks, defenses, and metrics. Moreover, we plan to open-source all the code and data of {\sc \small TrojanZoo}\xspace and encourage the community to contribute.

Second, to conduct unified evaluation, we mainly consider the attack vector of re-using pre-trained trojan models. There are other attack vectors through which backdoor attacks can be launched, including poisoning victims' training data\mcite{Shafahi:2018:nips,Zhu:2019:icml} and knowledge distillation\mcite{distillation-backdoor}, which entail additional constraints for attacks or defenses. For instance, the poisoning data needs to be evasive to bypass inspection. We consider studying alternative attack vectors as our ongoing work.

Finally, because of the plethora of work on neural backdoors in the computer vision domain, {\sc \small TrojanZoo}\xspace focuses on the image classification task, while recent work has also explored neural backdoors in other settings, including natural language processing\mcite{schuster-humpty,acl-backdoor,lm-trojan}, reinforcement learning\mcite{trojdrl}, and federated learning\mcite{federated-backdoor,dba}. We plan to extend {\sc \small TrojanZoo}\xspace to support such settings in its future releases.

\subsection{Additional Related Work}

Recent studies have surveyed neural backdoors\mcite{survey1,survey2,survey3}. However, none of them provides reference implementation or conducts empirical evaluation. Compared with the rich collection of platforms for adversarial attacks/defenses (\textit{e.g.}\xspace, {\sc CleverHans}\mcite{cleverhans}, {\sc DeepSec}\mcite{Ling:2019:sp}, and {\sc AdvBox}\mcite{advbox}), only few platforms currently support evaluating neural backdoors: {\sc Art}\mcite{ibm-art} integrates 3 attacks and 3 defenses, while {\sc TrojAI}\mcite{trojai} implements 1 attack and 3 metrics.

In comparison, {\sc \small TrojanZoo}\xspace differs in major aspects: \mcounter{i} to our best knowledge, it features the most comprehensive library of attacks/defenses; \mcounter{ii} it regards the evaluation metrics as a first-class citizen and implements 6 attack performance metrics and 10 defense utility metrics, which holistically assess given attacks/defenses; \mcounter{iii} besides reference implementation, it also provides rich utility tools for in-depth analysis of attack-defense interactions, such as measuring feature-space similarity, tracing neural activation patterns, and comparing attribution maps.

\section{Assessment}
\label{sec:evaluation}

Equipped with {\sc \small TrojanZoo}\xspace, we conduct a systematic assessment of the existing attacks and defenses, in which the attacks are evaluated in terms of effectiveness, evasiveness, and transferability, and the defenses are evaluated in terms of robustness, utility-preservation, and genericity.

\subsection{Experimental Setting}

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c|c|c|c|c}
{\bf Dataset} & {\bf \#\,Class} & {\bf \#\,Dimension} & {\bf Model} & {\bf Accuracy} \\
\hline
\hline
\multirow{3}{*}{CIFAR10} & \multirow{3}{*}{10} & \multirow{3}{*}{32$\times$32}  & ResNet18 & 95.37\% \\
\cline{4-5}
& & & DenseNet121 & 93.84\% \\
\cline{4-5}
& & &  VGG13 & 92.44\% \\
\hline
CIFAR100 & 100 &  32$\times$32   & \multirow{4}{*}{ResNet18} & 73.97\% \\
\cline{1-3}
\cline{5-5}
GTSRB & 43 & 32$\times$32  &  & 98.18\% \\
\cline{1-3}
\cline{5-5}
ImageNet-mini & 10 & 224$\times$224  & & 92.40\% \\
\cline{1-3}
\cline{5-5}
VGGFace2-mini & 20 & 224$\times$224  &  & 90.77\% \\
\end{tabular}
\caption{\footnotesize {\em \small ACC}\xspace of systems built upon benign, pre-trained models. \label{tab:dataset}}}
\end{table}

{\em \underline{Datasets} --} In the evaluation, we primarily use 5 datasets: {{CIFAR10}\xspace}\mcite{cifar}, {{CIFAR100}\xspace}\mcite{cifar}, {{ImageNet}\xspace}\mcite{imagenet},  {{GTSRB}\xspace}\mcite{gtsrb}, and {{VGGFace2}\xspace}\mcite{vggface2}. Their statistics are summarized in Table\mref{tab:dataset}. By default, we partition each dataset into 40\%/40\%/20\% for pre-training, fine-tuning, and testing respectively.

\vspace{1pt}
{\em \underline{Models} --} We consider 3 representative DNNs: VGG13\mcite{vgg}, ResNet18\mcite{resnet}, and DenseNet121\mcite{densenet}. Using models of distinct architectures (\textit{e.g.}\xspace, residual block), we factor out the influence of individual model characteristics. By default, we assume a downstream model comprising one fully-connected layer with softmax activation (1{\sc Fcn}). We also consider other types of models, including Bayes, SVM, and Random Forest. The {\em \small ACC}\xspace of systems built upon benign, pre-trained models is summarized in Table\mref{tab:dataset}.

\vspace{1pt}
{\em \underline{Attacks, Defenses, and Metrics} --} In the evaluation, we exemplify with 8 attacks in Table\mref{tab:attack_summary} and 12 defenses in Table\mref{tab:defense_summary}, and measure them using all the metrics in \msec{sec:attack-metric} and \msec{sec:defense-metric}. In all the experiments, we generate 10 trojan models for a given attack under each setting and 100 pairs of clean-trigger inputs with respect to each trojan model. The reported results are averaged over these cases.

\vspace{1pt}
{\em \underline{Implementation} --} All the models, algorithms, and measurements are implemented in PyTorch. All the experiments are conducted on a Linux server with Quodro RTX 6000 GPU, Intel Xeon processor, and 384G RAM. The default parameter setting is summarized as Table\mref{tab:attack_param} and\mref{tab:defense_param} in Appendix\mref{sec:details}.

\subsection{Attack Performance Evaluation}
\label{sec:attack-eval}

We first evaluate existing attacks on vanilla systems (without defenses), aiming to understand the impact of various design choices and context settings on attack performance. Due to space limitations, we mainly report the results on {CIFAR10}\xspace and defer the results on other datasets to Appendix\mref{sec:additional}.

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/asr-size-cifar.eps, width = 85mm}
    \caption{\footnotesize {\em \small ASR}\xspace and {\em \small TMC}\xspace with respect to trigger size ($\alpha$\,=\,0.8, {CIFAR10}\xspace). \label{fig:size_cifar10}}
\end{figure}

\vspace{1pt}
{\bf Trigger size --} Recall that the trigger definition consists of mask\,$m$, transparency\,$\alpha$, and pattern\,$\phi$. Here, we measure how the attack efficacy varies with the trigger size\,$|m|$. To make fair comparison, we bound the clean accuracy drop ({\em \small CAD}\xspace) of all the attacks below 3\% via controlling the number of optimization iterations\,$n_{\mathrm{iter}}$. Figure\mref{fig:size_cifar10} plots the attack success rate ({\em \small ASR}\xspace) and trojan misclassification confidence ({\em \small TMC}\xspace) of various attacks under varying $|m|$ on {CIFAR10}\xspace (with fixed $\alpha = 0.8$).

Observe that most attacks seem insensitive to $|m|$: as $|m|$ varies from 2$\times$2 to 5$\times$5, the {\em \small ASR}\xspace of most attacks increases by less than 10\%, except {\sc Rb}\xspace and {\sc Tb}\xspace, with over 30\% growth. This may be attributed to their additional constraints: {\sc Rb}\xspace defines the trigger to be the reflection of another image, while {\sc Tb}\xspace requires the trigger to be positionless). Thus, increasing $|m|$ may improve their perturbation spaces.
Also observe that the {\em \small TMC}\xspace of most attacks remains close to 1.0 regardless of $|m|$.

\begin{mtbox}{\small Remark\,1}
    {\em \small Trigger-size has a limited impact on attack efficacy, except for attacks with additional trigger constraints.}
\end{mtbox}

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/asr-alpha-cifar.eps, width = 82mm}
    \caption{\footnotesize {\em \small ASR}\xspace with respect to trigger transparency ($|m|$ = 3$\times$3, {CIFAR10}\xspace). \label{fig:alpha_cifar10}}
\end{figure}

{\bf Trigger transparency --}  Under the same setting, we further evaluate the impact of trigger transparency\,$\alpha$. Figure\mref{fig:alpha_cifar10} plots the {\em \small ASR}\xspace of different attacks as a function of $\alpha$ on {CIFAR10}\xspace (with fixed $|m|$ = 3$\times$3).

Compared with trigger size, $\alpha$ has a more profound impact. The {\em \small ASR}\xspace of most attacks drops sharply once $\alpha$ exceeds 0.6, among which {\sc Tb}\xspace approaches 10\% if $\alpha \geq 0.8$, and {\sc Esb}\xspace works only if $\alpha$ is close to 0, due to its reliance on recognizing the trigger precisely to overwrite the model prediction. Meanwhile, {\sc Lb}\xspace and {\sc Imc}\xspace seem insensitive to $\alpha$. This may be attributed to that {\sc Lb}\xspace optimizes trojan models with respect to latent representations (rather than final predictions), while {\sc Imc}\xspace optimizes trigger patterns and trojan models jointly. Both strategies may mitigate $\alpha$'s impact.

\begin{mtbox}{\small Remark\,2}
{\em \small It requires to exploit alternative optimization strategies to attain effective attacks under high trigger-transparency.}
\end{mtbox}

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/asr-cad-auc.eps, width = 82mm}
    \caption{\footnotesize Attack efficacy-specificity trade-off on {CIFAR10}\xspace ($|m|\,=\,3\times3$, $\alpha$\,=\,0.8) \label{fig:auc}}
\end{figure}

\vspace{1pt}
{\bf Efficacy-specificity trade-off --} One intriguing property of attacks is the trade-off between maximizing the effectiveness with respect to trigger inputs (efficacy) and minimizing the influence over clean inputs (specificity). Here, we characterize the efficacy-specificity trade-off via varying the fraction of trigger inputs in training data. For each attack, we bound its {\em \small CAD}\xspace within 3\%, measure its highest and lowest {\em \small ASR}\xspace (which corresponds to its lowest and highest {\em \small CAD}\xspace respectively), and then normalize the {\em \small ASR}\xspace and {\em \small CAD}\xspace measures to $[0, 1]$.

Figure\mref{fig:auc} visualizes the normalized {\em \small CAD}\xspace-{\em \small ASR}\xspace trade-off. Observe that the curves of all the attacks manifest strong convexity, indicating the ``leverage'' effects\mcite{imc}: it is practical to greatly improve {\em \small ASR}\xspace at a disproportionally small cost of {\em \small CAD}\xspace. Also observe that different attacks feature varying
Area Under the Curve ({\em \small AUC}\xspace). Intuitively, a smaller {\em \small AUC}\xspace implies a stronger leverage effect. Among all the attacks, {\sc Imc}\xspace shows the smallest {\em \small AUC}\xspace. This may be explained by that {\sc Imc}\xspace uses the trigger-model co-optimization framework, which allows the adversary to maximally optimize {\em \small ASR}\xspace at given {\em \small CAD}\xspace.

\begin{mtbox}{\small Remark\,3}
{\em \small All the attacks demonstrate strong ``leverage'' effects in the efficacy-specificity trade-offs.}
\end{mtbox}

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{2pt}
    \begin{tabular}{c|c|c|c|c}
        \multirow{2}{*}{\bf Attack} & {\bf CIFAR10}  & {\bf CIFAR100}    & \multicolumn{2}{c}{\bf ImageNet}  \\
        \cline{2-5}
        & $|m|$\,=\,3, $\alpha$\,=\,0.8 & $|m|$\,=\,3, $\alpha$\,=\,0.8 & $|m|$\,=\,3, $\alpha$\,=\,0 & $|m|$\,=\,7, $\alpha$\,=\,0.8 \\
        \hline
        \hline
        {\sc Bn}\xspace     & 72.4 (0.96) & 64.5 (0.96) & 90.0 (0.98) &     11.4 (0.56)       \\
        {\sc Tnn}\xspace    & 91.5 (0.97) & 89.8 (0.98) & 95.2 (0.99) &   11.6 (0.62)         \\
        {\sc Rb}\xspace     & 52.1 (1.0) & 42.8 (0.95) & 94.6 (0.98) &    11.2 (0.59)        \\
        {\sc Tb}\xspace     & 11.5 (0.66) & 23.4 (0.75) & 82.8 (0.97) &    11.4 (0.58)        \\
        {\sc Lb}\xspace     & \cellcolor{Red}100.0 (1.0) & 97.8 (0.99) & 97.4 (0.99) & 11.4 (0.59) \\
        {\sc Esb}\xspace    & 10.3 (0.43) & 1.0 (0.72) & \cellcolor{Red}100.0 (0.50) &    N/A         \\
        {\sc Abe}\xspace    & 74.3 (0.91) & 67.9 (0.96) & 82.6 (0.97) &    12.00 (0.50)           \\
        {\sc Imc}\xspace    & \cellcolor{Red}100.0 (1.0) & \cellcolor{Red}98.8 (0.99) & 98.4 (1.0) &  \cellcolor{Red}96.6 (0.99)       \\
    \end{tabular}
    \caption{\footnotesize Impact of data complexity on {\em \small ASR}\xspace and {\em \small TMC}\xspace of various attacks. \label{tab:data-comprison}}}
\end{table}

\vspace{1pt}
{\bf Data complexity --} To assess the impact of data complexity, we compare the {\em \small ASR}\xspace and {\em \small TMC}\xspace of existing attacks on different datasets, with results in Table\mref{tab:data-comprison} (more results in Table\mref{tab:data-comprison2}).

We observe that the class-space size (the number of classes) negatively affects the attack efficacy. For example, the {\em \small ASR}\xspace of {\sc Bn}\xspace drops by 7.9\% from {CIFAR10}\xspace to {CIFAR100}\xspace. Intuitively, it is more difficult to force trigger inputs from all the classes to be misclassified in a larger output-space. Moreover, it tends to require more significant triggers to achieve comparable attack performance on more complex data. For instance, for {\sc Imc}\xspace to attain similar {\em \small ASR}\xspace on {CIFAR10}\xspace and  {ImageNet}\xspace, it needs to either increase trigger size (from 3$\times$3 to 7$\times$7) or reduce trigger transparency (from 0.8 to 0.0).
\begin{mtbox}{\small Remark\,4}
{\em \small To attain comparable attack efficacy on more complex data requires more significant triggers.}
\end{mtbox}

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/model-arch-bar.eps, width = 82mm}
    \caption{Impact of DNN architecture on attack efficacy. \label{fig:model_arch}}
\end{figure}

\vspace{1pt}
{\bf DNN architecture --} Another important factor that may impact attack efficacy is DNN architecture. Here, we compare the performance of various attacks on three DNN model, VGG, ResNet, and DenseNet, with results shown in Figure\mref{fig:model_arch}.

We have the following observations. First, different model architectures manifest varying attack vulnerabilities, ranked as ResNet\,$>$\,DenseNet\,$>$\,VGG. This may be explained as follows. Compared with traditional convolutional networks (\textit{e.g.}\xspace, VGG), the unique constructs of ResNet (\textit{i.e.}\xspace,\,residual block) and DenseNet (\textit{i.e.}\xspace,\,skip connection) enable more effective feature extraction, but also allow more effective propagation of trigger patterns. Second, among all the attacks, {\sc Lb}\xspace, {\sc Imc}\xspace, and {\sc Esb}\xspace seem insensitive to model architectures, which may be attributed to the optimization strategies of {\sc Lb}\xspace and {\sc Imc}\xspace, and the direct  modification of DNN architectures by {\sc Esb}\xspace.

\begin{mtbox}{\small Remark\,5}
{\em \small DNN architectures that enable better feature extraction may also allow more effective propagation of trigger patterns.}
\end{mtbox}

\begin{table}[!ht]
    {\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
  \setlength{\tabcolsep}{4pt}
    \begin{tabular}{r|c|c|c|c|c|c|c}
    \multirow{2}{*}{\bf Attack} & \multicolumn{3}{c|}{\bf Fine-Tuning} & \multicolumn{4}{c}{\bf Downstream Classifier} \\
    \cline{2-8}
    & {\bf None} & {\bf Partial} & {\bf Full} & {\bf 2-FCN} & {\bf Bayes} & {\bf SVM}  & {\bf RF}\\
    \hline
    \hline
    {\sc Bn}\xspace & 72.4 & 72.3 & 30.4 &  72.2 &  73.5 & 64.7 & 66.0 \\
    {\sc Tnn}\xspace & 91.5 &  89.6 & 27.1 &  90.8 & 90.3 & 82.9 & 81.1 \\
    {\sc Rb}\xspace & 79.2 & 77.0 & 12.4 &  78.3 & 76.8 & 61.5 & 63.7 \\
    {\sc Lb}\xspace & {\cellcolor{Red}}100.0 & {\cellcolor{Red}}100.0 &  {\cellcolor{Red}}95.3 &  {\cellcolor{Red}}99.9 & 99.9 &  {\cellcolor{Red}}99.9 &  {\cellcolor{Red}}99.8 \\
    {\sc Imc}\xspace & {\cellcolor{Red}}100.0 &  99.9 & 88.7 &   {\cellcolor{Red}}99.9 &  {\cellcolor{Red}}100.0 & {\cellcolor{Red}}99.9 &  {\cellcolor{Red}}99.8 \\

    \end{tabular}
    \caption{\footnotesize Impact of fine-tuning and downstream model on attack efficacy.\label{tab:fine-tune} }}
\end{table}

\vspace{1pt}
{\bf Fine-tuning and downstream model --} Recall that a trojan model is often composed with a downstream model and fine-tuned for the target task. We evaluate the impact of downstream-model selection and fine-tuning strategy on the attack efficacy. We consider 5 different downstream models (1/2 fully-connected layer, Bayes, SVM, and Random Forest) and 3 fine-tuning strategies (none, partial tuning, and full tuning). Note that the adversary is unaware of such settings.

Table\mref{tab:fine-tune} compares the performance of 5 attacks with respect to varying downstream models and fine-tuning methods. Observe that fine-tuning has a great impact on attack efficacy. For instance, the {\em \small ASR}\xspace of {\sc Tnn}\xspace drops by 62.5\% from partial- to full-tuning. In comparison, {\sc Lb}\xspace and {\sc Imc}\xspace are less subjective to fine-tuning, due to their optimization strategies. Also observe that the attack performance seems agnostic to the downstream model. This may be explained by that the downstream model in practice tends to manifest ``pseudo-linearity''\mcite{Ji:2018:ccsa}, making the system's output linearly correlated with the trojan model's output (more details in Appendix\mref{sec:details}).

\begin{mtbox}{\small Remark\,6}
{\em \small The performance of most attacks is subjective to fine-tuning strategies but agnostic to downstream-model selections.}
\end{mtbox}

\vspace{1pt}
{\bf Transferability --} Next, we consider the setting that without access to the data from the downstream task, the adversary pre-trains the trojan model on a surrogate dataset and transfers the attack to the target dataset.

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{1pt}
\begin{tabular}{c|c|c|c|c|c|c}

\multicolumn{2}{c|}{\bf Transfer Setting} & \multicolumn{5}{c}{\bf Attack} \\
\hline
{\bf Surrogate} & {\bf Target} & {\sc Bn}\xspace  & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Lb}\xspace & {\sc Imc}\xspace \\
\hline
\hline
\multirow{2}{*}{{CIFAR10}\xspace} & {CIFAR10}\xspace & 94.5\,(0.99)  &{\cellcolor{Red}}100.0\,(1.0) &{\cellcolor{Red}}100.0\,(1.0) & {\cellcolor{Red}}100.0\,(1.0) & {\cellcolor{Red}}100.0\,(1.0) \\
& {ImageNet}\xspace & 8.4\,(0.29)   & 7.8\,(0.29)  & 8.6\,(0.30)   & 8.2\,(0.30)    &  \cellcolor{Red}9.4\,(0.32)    \\
\hline
\multirow{2}{*}{{ImageNet}\xspace} & {ImageNet}\xspace  & 90.0\,(0.98)  & 95.2\,(0.99) & 94.6\,(0.98) & 97.4\,(0.99)  &{\cellcolor{Red}}98.4\,(1.0)  \\
& {CIFAR10}\xspace &{\cellcolor{Red}}77.0\,(0.84)) & 26.9\,(0.72)         & 11.0\,(0.38)         & 10.0\,(0.38)          & 14.3\,(0.48)          \\

\end{tabular}
\caption{\footnotesize {\em \small ASR}\xspace and {\em \small TMC}\xspace of transfer attacks across {CIFAR10}\xspace and {ImageNet}\xspace ($|m|$\,=\,3$\times$3, $\alpha$\,=\,0.0). \label{tab:transfer}}}
\end{table}

We evaluate the efficacy of transferring attacks across two datasets, {CIFAR10}\xspace and {ImageNet}\xspace, with results summarized in Table\mref{tab:transfer}. We have the following findings. Several attacks (\textit{e.g.}\xspace, {\sc Bn}\xspace) are able to transfer from {ImageNet}\xspace to {CIFAR10}\xspace to a certain extent, but most attacks fail to transfer from {CIFAR10}\xspace to {ImageNet}\xspace. The finding may be justified as follows. A model pre-trained on complex data (\textit{i.e.}\xspace, {ImageNet}\xspace) tends to maintain its effectiveness of feature extraction on simple data (\textit{i.e.}\xspace, {CIFAR10}\xspace)\mcite{Esteva:2017:nature}; as a side effect, it may also preserve its effectiveness of propagating trigger patterns. Meanwhile, a model pre-trained on simple data may not generalize well to complex data. Moreover, compared with stronger attacks in non-transfer cases (\textit{e.g.}\xspace, {\sc Lb}\xspace), {\sc Bn}\xspace shows much higher transferability. This may be explained by that to maximize the attack efficacy, the trigger and trojan model often need to ``over-fit'' the training data, resulting in their poor transferability.

\begin{mtbox}{\small Remark\,7}
{\em \small Backdoor attacks tend to transfer from complex data to simple data but not vice versa, while weak attacks demonstrate higher transferability than strong ones.}
\end{mtbox}

\subsection{Defense Utility Evaluation}
\label{sec:defense-eval}

Next, we evaluate the utility preservation of defenses, to understand their impact on the system's normal functionality.

\vspace{1pt}
{\bf Clean accuracy --} We first measure the impact of defenses on the accuracy of classifying clean inputs. As input filtering and model inspection have no direct influence on clean accuracy, we focus on input transformation and model sanitization.

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{c|c|c|c|c|c|c|c|c|c}

\multirow{2}{*}{\bf Defense} & \multicolumn{9}{c}{\bf Attack} \\
\cline{2-10}
& -- & {\sc Bn}\xspace   & {\sc Tnn}\xspace  & {\sc Rb}\xspace  & {\sc Tb}\xspace   & {\sc Lb}\xspace   & {\sc Esb}\xspace  & {\sc Abe}\xspace  & {\sc Imc}\xspace  \\
\hline
\hline
-- & 95.4 & 95.3 & 95.2 & 95.4 & 95.3 & 95.5 & 95.3 & 95.0 & 95.5 \\
\hline
{\sc Rs}\xspace     & -0.3 & -0.6 & -0.3 & -0.4 & -0.4 & -0.3 & -0.3 & -0.4 & -0.5 \\
{\sc Du}\xspace        & -4.0 & -4.5 & -4.5 & -4.4 & -4.3 & -4.3 & -4.0 & -4.9 & -4.6 \\
{\sc Mp}\xspace       & \cellcolor{Red}-11.2 & \cellcolor{Red}-11.9 &\cellcolor{Red}-11.3 & \cellcolor{Red}-10.8 & \cellcolor{Red}-11.3 & \cellcolor{Red}-11.4 & \cellcolor{Red}-11.2 & \cellcolor{Red}-11.9 & \cellcolor{Red}-11.0 \\
\hline
{\sc Fp}\xspace        & -0.1 & -0.2 & +0.0 & +0.0 & +0.0 &-0.2 & -0.2 & +0.3 & -0.4 \\
{\sc Ar}\xspace        & \cellcolor{Red}-11.1 & \cellcolor{Red}-11.1 & \cellcolor{Red}-10.4 & \cellcolor{Red}-10.4 & \cellcolor{Red}-10.4 & \cellcolor{Red}-10.9 & \cellcolor{Red}-10.9 & \cellcolor{Red}-10.5 & \cellcolor{Red}-11.4 \\
\end{tabular}
\caption{\footnotesize Impact of defenses on classification accuracy of {CIFAR10}\xspace ($-$: clean model without attack/defense). \label{tab:clean-acc}}}
\end{table}

Table\mref{tab:clean-acc} summarizes the results. With the no-defense setting as the baseline, most defenses tend to negatively affect clean accuracy, yet with varying impact. For instance, across all the cases, {\sc Rs}\xspace and {\sc Ar}\xspace cause about 0.4\% and 11\% {\em \small CAD}\xspace respectively. This is explained by the difference of their underlying mechanisms: although both attempt to alleviate the influence of trigger patterns, {\sc Rs}\xspace smooths the prediction of an input $x$ over its vicinity, while {\sc Ar}\xspace forces the model to make consistent predictions in $x$'s vicinity. Also note that {\sc Fp}\xspace attains the least {\em \small CAD}\xspace across all the cases, mainly due to its fine-tuning.

\begin{mtbox}{\small Remark\,8}
{\em \small Input-transformation and model-sanitization negatively impact accuracy, while fine-tuning may mitigate such effect.}
\end{mtbox}

\vspace{1pt}
{\bf Execution Time  --} We compare the overhead of various defenses by measuring their {\em \small ART}\xspace (\msec{sec:defense-metric}). The results are listed in Table\mref{tab:time}. Note that online defenses (\textit{e.g.}\xspace, {\sc Strip}\xspace) have negligible overhead, while offline methods (\textit{e.g.}\xspace, {\sc Abs}\xspace) require longer but acceptable running time (10$^3$$\sim$10$^4$ seconds).

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{3pt}
    \begin{tabular}{c|c|c|c|c}
   {\sc Mp}\xspace &  {\sc Neo}\xspace  & {\sc Strip}\xspace &  {\sc Ar}\xspace &  {\sc Fp}\xspace \\
   \hline
  2.4$\times$10$^1$  &   7.7$\times$10$^0$    &   1.8$\times$10$^{-1}$  &  1.7$\times$10$^4$  & 2.1$\times$10$^3$ \\
  \hline
  \hline

   {\sc Nc}\xspace & {\sc Tabor}\xspace & {\sc Abs}\xspace & {\sc Ni}\xspace & {\sc Di}\xspace  \\
   \hline
 1.8$\times$10$^3$ & 4.2$\times$10$^3$  & 1.9$\times$10$^3$ & 4.6$\times$10$^1$ & 4.1$\times$10$^2$
    \end{tabular}
    \caption{\footnotesize Running time of various defenses (second). \label{tab:time}}}
\end{table}

\begin{mtbox}{\small Remark\,9}
{\em \small Most defenses have marginal execution overhead with respect to practical datasets and models.}
\end{mtbox}

\begin{figure*}[!ht]
    \centering
    \epsfig{file=figures/defense/defense-trigger-inference.eps, width=140mm}
    \caption{\footnotesize TPR of {\sc Neo}\xspace and {\sc Strip}\xspace under varying trigger definition (left:  $|m|=3\times3$, right:  $|m|=6\times6$; lower:  $\alpha=0.0$, upper:  $\alpha=0.8$). \label{fig:inference-trigger}}
\end{figure*}

\subsection{Attack-Defense Interaction Evaluation}
\label{sec:attack-defense-eval}

In this set of experiments, we evaluate the robustness of existing defenses with respect to various attacks, aiming to characterize their dynamic interactions. As the defenses from different categories bear distinct objectives (\textit{e.g.}\xspace, detecting trigger inputs versus cleansing trojan models). we evaluate each defense category separately.

\vspace{1pt}
{\bf Attack-agnostic defenses --} Input transformation and model sanitization mitigate backdoors in an attack-agnostic manner. We measure their robustness using {\em \small ARD}\xspace and {\em \small TMC}\xspace.

\begin{table*}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \begin{tabular}{c|c|c|c|c|c|c|c|c}
        \multirow{2}{*}{\bf Defense} & \multicolumn{8}{c}{\bf Attack} \\
        \cline{2-9}
             & {\sc Bn}\xspace           & {\sc Tnn}\xspace          & {\sc Rb}\xspace          & {\sc Tb}\xspace           & {\sc Lb}\xspace           & {\sc Esb}\xspace          & {\sc Abe}\xspace          & {\sc Imc}\xspace          \\
        \hline
        \hline
        -- & 93.3 (0.99) & 99.9 (1.0) & 99.8 (1.0) & 96.7 (0.99) & 100.0 (1.0) & 100.0 (0.86) & 95.3 (0.99) & 100.0 (1.0) \\
        \hline
        {\sc Rs}\xspace      & -0.5 (0.99) & -0.0 (1.0) & -0.0 -(1.0) & -0.3 (0.99) & -0.0 (1.0) & -89.1 (0.86) & -0.5 (0.99) & -0.0 (1.0) \\
        {\sc Du}\xspace         & -2.2 (0.99) & -0.4 (1.0) & -5.4 (1.0) & \cellcolor{Red}-67.8 (1.0) & -4.1 (1.0) & \cellcolor{Red}-89.9 (0.86) & -0.5 (0.99) & -0.2 (1.0)         \\
        {\sc Mp}\xspace        & \cellcolor{Red}-6.0 (0.99) & \cellcolor{Red}-37.4 (1.0) & \cellcolor{Red}-78.6 (1.0) & -11.0 (0.99) & \cellcolor{Red} -42.6 (1.0) & -87.8 (0.86) & \cellcolor{Red}-4.6 (0.99) & \cellcolor{Red}-16.0 (1.0) \\
        \hline
        {\sc Fp}\xspace         & -82.9 (0.60) & -86.5 (0.64) & -89.1 (0.73) & -38.0 (0.89) & -27.6 (0.82) & \cellcolor{Red}-100.0 (0.81) &-84.5 (0.64) & -26.9 (0.83) \\
        {\sc Ar}\xspace         & \cellcolor{Red}-83.2 (0.84) & \cellcolor{Red}-89.6 (0.85) & \cellcolor{Red}-89.8 (0.62) & \cellcolor{Red}-86.2 (0.63) & \cellcolor{Red}-90.1 (0.83) & -100.0 (0.86) & \cellcolor{Red}-85.3 (0.81) & \cellcolor{Red}-89.7 (0.83) \\
    \end{tabular}
    \caption{\footnotesize {\em \small ARD}\xspace and {\em \small TMC}\xspace of attack-agnostic defenses against various attacks. \label{tab:attack-agnostic}}}
\end{table*}

With the no-defense case as reference, Table\mref{tab:attack-agnostic} compares the robustness of various defenses, with the following findings: \mcounter{i} {\sc Mp}\xspace and {\sc Ar}\xspace are the most robust methods in the categories of input transformation and model sanitization, respectively, which however are attained with over 10\% {\em \small CAD}\xspace (\textit{cf}.\xspace Table\mref{tab:clean-acc}). \mcounter{ii}
{\sc Fp}\xspace seems effective against most attacks except {\sc Lb}\xspace and {\sc Imc}\xspace, which is explained as follows: unlike attacks (\textit{e.g.}\xspace, {\sc Tnn}\xspace) that optimize the trigger with respect to selected neurons, {\sc Lb}\xspace and {\sc Imc}\xspace perform optimization with respect to all the neurons,  making them immune to the pruning of {\sc Fp}\xspace.
\mcounter{iii} Most defenses are able to defend against {\sc Esb}\xspace (over 85\% {\em \small ARD}\xspace), which is attributed to its hard-coded trigger pattern and modified DNN architecture: slight perturbation to the trigger input or trojan model may destroy the embedded backdoor.

\begin{mtbox}{\small Remark\,10}
{\em \small In model sanitization or input transformation, there exists an accuracy-robustness trade-off.}
\end{mtbox}

\vspace{1pt}
{\bf Input filtering --} Next, we evaluate the robustness of input filtering defenses. With respect to each attack, we randomly generate 100 pairs of trigger-clean inputs and measure the {\em \small TPR}\xspace and {\em \small FPR}\xspace of {\sc Strip}\xspace and {\sc Neo}\xspace, two representative input filtering methods. To make easy comparison, we fix {\em \small FPR}\xspace as 0.05 and report {\em \small TPR}\xspace in Table\mref{tab:input-filtering} (more statistics in Appendix\mref{sec:additional}).

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
 \setlength{\tabcolsep}{4pt}
    \begin{tabular}{c|c|c|c|c|c|c|c|c}
        \multirow{2}{*}{\bf Defense} & \multicolumn{8}{c}{\bf Attack} \\
        \cline{2-9}
    & {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Tb}\xspace & {\sc Lb}\xspace & {\sc Esb}\xspace & {\sc Abe}\xspace & {\sc Imc}\xspace \\
    \hline
    \hline
   {\sc Strip}\xspace & 0.07 & 0.13 & 0.34 & 0.27 & \cellcolor{Red}0.91 & 0.10 & 0.07 & \cellcolor{Red}0.99 \\
  {\sc Neo}\xspace & 0.29 & 0.23 & 0.29 & 0.36 & 0.29 & \cellcolor{Red}0.64 & 0.28 & 0.29 \\
    \end{tabular}
    \caption{\footnotesize TPR of {\sc Neo}\xspace and {\sc Strip}\xspace (FPR\,=\,0.05, $\alpha$\,=\,0.0). \label{tab:input-filtering}}}
\end{table}

We have the following findings. \mcounter{i} {\sc Strip}\xspace is particularly robust against {\sc Lb}\xspace and {\sc Imc}\xspace (over 0.9 {\em \small TPR}\xspace). Recall that {\sc Strip}\xspace detects a trigger input using the self-entropy of its mixture with a clean input. This indicates that the triggers produced by {\sc Lb}\xspace and {\sc Imc}\xspace effectively dominate the mixtures, which is consistent with the findings in other experiments (\textit{cf}.\xspace Figure\mref{tab:attack_summary}). \mcounter{ii} {\sc Neo}\xspace is robust against most attacks to a limited extent (less than 0.3 {\em \small TPR}\xspace), but especially effective against {\sc Esb}\xspace (over 0.6 {\em \small TPR}\xspace), mainly due to its requirement for recognizing the trigger pattern precisely to overwrite the model prediction.

\begin{mtbox}{\small Remark\,11}
{\em \small Trigger design faces the effectiveness-evasiveness trade-off with respect to input-filtering defenses.}
\end{mtbox}

We also evaluate the impact of trigger definition on the performance of input filtering, with results in Figure\mref{fig:inference-trigger} (results for other defenses in Appendix\mref{sec:additional}). With fixed trigger transparency, {\sc Neo}\xspace constantly attains higher {\em \small TPR}\xspace under larger triggers; in comparison, {\sc Strip}\xspace seems less sensitive. This is attributed to the difference of their detection rationale: given an input $x$, {\sc Neo}\xspace searches for ``tipping'' position in $x$ to cause prediction change, which is clearly subjective to the trigger size; while {\sc Strip}\xspace measures the self-entropy of $x$'s mixture with a clean input, which does not rely on the trigger size.

\begin{mtbox}{\small Remark\,12}
{\em \small Trigger design also faces the trade-off between the evasiveness with respect to different input-filtering defenses.}
\end{mtbox}

\vspace{1pt}
{\bf Model inspection --} We evaluate model-inspection defenses in terms of their effectiveness of \mcounter{i} identifying trojan models and \mcounter{ii} recovering trigger patterns.

Specifically, given defense ${\mathcal{D}}$ and model $f$, we measure the {\em \small AIV}\xspace of all the classes; if $f$ is a trojan model, we use the {\em \small AIV}\xspace of the target class to quantify ${\mathcal{D}}$'s {\em \small TPR}\xspace of detecting trojan models and target classes; if $f$ is a clean model, we use the largest {\em \small AIV}\xspace to quantify ${\mathcal{D}}$'s {\em \small FPR}\xspace of misclassifying clean models. The results are summarized in Table\mref{tab:anidx}.

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{2pt}
    \begin{tabular}{c|c|c|c|c|c|c|c|c}
    & {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Tb}\xspace & {\sc Lb}\xspace & {\sc Esb}\xspace & {\sc Abe}\xspace & {\sc Imc}\xspace\\
    \hline
    \hline
    {\sc Nc}\xspace & 3.08 &  2.69 & 2.48 & 2.44 &2.12 & 0.04 & 2.67 & 1.66 \\
    {\sc Di}\xspace & 0.54 & 0.46 & 0.39 & 0.29 & 0.21 & 0.01 & 0.76 & 0.26 \\
    {\sc Tabor}\xspace & \cellcolor{Red}3.26 &2.49 &  2.32 &2.15 & 2.01 & 0.89 &  2.44 & 1.89 \\
    {\sc Ni}\xspace & 1.28 & 0.59 & 0.78 & 1.11 & 0.86 & 0.71 & 0.41 & 0.52 \\
    {\sc Abs}\xspace &  3.02 & \cellcolor{Red}4.16 & \cellcolor{Red}4.10 &  \cellcolor{Red}15.55 &  \cellcolor{Red}2.88 & \cellcolor{Gray} &  \cellcolor{Red}8.45 & \cellcolor{Red}3.15 \\
    \end{tabular}
    \caption{\footnotesize {\em \small AIV}\xspace of clean models and trojan models by various attacks. \label{tab:anidx}}}
\end{table}
 \mcounter{i} Compared with other defenses, {\sc Abs}\xspace is highly effective in detecting trojan models (with largest {\em \small AIV}\xspace), attributed to its neuron sifting strategy. \mcounter{ii} {\sc Imc}\xspace seems evasive to most defenses (with {\em \small AIV}\xspace below 2). This is explained by its trigger-model co-optimization strategy, which minimizes model distortion. \mcounter{iii} Most model-inspection defenses are either ineffective or inapplicable against {\sc Esb}\xspace, as it keeps the original DNN intact but adds an additional module. This contrasts the high effectiveness of other defenses against {\sc Esb}\xspace (\textit{cf}.\xspace Table\mref{tab:attack-agnostic}).

\begin{mtbox}{\small Remark\,13}
    {\em \small There exists a trade-off among the evasiveness with respect to model inspection and other defenses.}
\end{mtbox}

\begin{table*}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \begin{tabular}{c|c c|c c|c c|c c|c c|c c|c c|c c}
        \multirow{3}{*}{\bf Defense} & \multicolumn{16}{c}{\bf Attack} \\
        \cline{2-17}
       & \multicolumn{2}{c|}{{\sc Bn}\xspace}    & \multicolumn{2}{c|}{{\sc Tnn}\xspace}     & \multicolumn{2}{c|}{{\sc Rb}\xspace}   & \multicolumn{2}{c|}{{\sc Tb}\xspace}  & \multicolumn{2}{c|}{{\sc Lb}\xspace}  & \multicolumn{2}{c|}{{\sc Esb}\xspace}       & \multicolumn{2}{c|}{{\sc Abe}\xspace}         & \multicolumn{2}{c}{{\sc Imc}\xspace}      \\
       \cline{2-17}
       & MLN & MJS & MLN & MJS & MLN & MJS & MLN & MJS & MLN & MJS & MLN & MJS & MLN & MJS & MLN & MJS \\
        \hline
        \hline
        {\sc Nc}\xspace  &  \cellcolor{Blue}4.98  & 0.55   &   \cellcolor{Blue}4.65 & \cellcolor{Red}0.70      &  \cellcolor{Blue}2.64 & \cellcolor{Red}0.89   &  \cellcolor{Blue}3.53 & \cellcolor{Gray}    &  \cellcolor{Blue}7.52 & 0.21  & 35.16 & 0.00  & 5.84 & \cellcolor{Red}0.42         &  \cellcolor{Blue}8.63 & 0.13      \\
        {\sc Di}\xspace  &  9.65 & 0.25   & 6.88 & 0.17     & 4.77 & 0.30  & 8.44 & \cellcolor{Gray}  & 20.17 & 0.21 &  \cellcolor{Blue}0.00 & \cellcolor{Red}0.06  & 10.21 & 0.30         & 12.78 & 0.25       \\
        {\sc Tabor}\xspace & 5.63 & \cellcolor{Red}0.70   & 4.47 & 0.42     & 3.03 & 0.70  & 3.67 & \cellcolor{Gray}  & 7.65 & 0.21 & 43.37 & 0.00  &  \cellcolor{Blue}5.65 & \cellcolor{Red}0.42         & 8.69 & 0.13      \\
        {\sc Abs}\xspace   &  17.74 & 0.42   &  17.91 & 0.55     &  17.60 & 0.70  &  16.00 & \cellcolor{Gray} &  17.29 & \cellcolor{Red}0.42 & \cellcolor{Gray}  &\cellcolor{Gray}   &  17.46 & 0.31  &  17.67 & \cellcolor{Red}0.31     \\
    \end{tabular}
    \caption{\footnotesize {\em \small MLN}\xspace and {\em \small MJS}\xspace of triggers recovered by model-inspection defenses with respect to various attacks (Note: as the trigger position is randomly chosen in {\sc Tb}\xspace, its {\em \small MJS}\xspace is un-defined). \label{tab:trigger-recover} }}
\end{table*}

For successfully detected trojan models, we further evaluate the trigger recovery of various defenses by measuring the mask $\ell_1$ norm ({\em \small MLN}\xspace) of recovered triggers and mask jaccard similarity ({\em \small MJS}\xspace) between recovered and injected triggers, with results shown in Table\mref{tab:trigger-recover}. While the ground-truth trigger has {\em \small MLN}\xspace\,=\,9 ($\alpha$\,=\,0.0, $|m|$\,=\,3$\times$3), most defenses recover triggers of varying {\em \small MLN}\xspace and non-zero {\em \small MJS}\xspace, indicating that they recover triggers different from, yet overlapping with, the injected ones. This finding is consistent with recent studies\mcite{mesa,broken-backdoor} that show a backdoor attack essentially injects a trigger distribution rather than a specific trigger.

\begin{mtbox}{\small Remark\,14}
{\em \small An attack, while targeting a specific trigger, essentially injects a trigger distribution; mode-inspection defenses tend to recover triggers related to, but non-identical to, the injected one.}
\end{mtbox}

\section{Exploration}
\label{sec:exploration}

Next, we examine the current practices of operating backdoor attacks and defenses and explore potential improvement.

\subsection{Attack -- Trigger}

We first explore improving the trigger definition by answering the following questions.

\vspace{1pt}
{\bf RQ$\bm{_1}$:\,{\em Is it necessary to use large triggers?}}

It is found in \msec{sec:attack-eval} that attack efficacy seems insensitive to trigger size. We now consider the extreme case that the trigger is defined as a single pixel and evaluate the efficacy of different attacks (constrained by {\em \small CAD}\xspace below 5\%), with results show in Table\mref{tab:one-pixel}. Note that the  trigger definition is inapplicable to {\sc Esb}\xspace, due to its requirement for trigger size.

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{c|c|c|c|c|c|c|c}
    {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Tb}\xspace & {\sc Lb}\xspace & {\sc Esb}\xspace & {\sc Abe}\xspace & {\sc Imc}\xspace \\
\hline
\hline
95.1 & 98.1 & 77.7 & 98.0 & \cellcolor{Red}100.0 & \cellcolor{Gray} & 90.0 & 99.7 \\
(0.99)  & (0.96) & (0.96) &(0.99) & \cellcolor{Red}(0.99) & \cellcolor{Gray} & (0.97) & (0.99)
\end{tabular}
\caption{\footnotesize {\em \small ASR}\xspace and {\em \small TMC}\xspace of single-pixel triggers ($\alpha$\,=\,0.0, {\em \small CAD}\xspace\,$\leq$\,5\%).\label{tab:one-pixel} }}
\end{table}

Surprisingly, with single-pixel triggers, most attacks achieve {\em \small ASR}\xspace comparable with the cases of larger triggers (\textit{cf}.\xspace Figure\mref{fig:size_cifar10}). This implies the existence of universal, single-pixel perturbation\mcite{universal-perturbation} with respect to trojan models (but not clean models!), highlighting the mutual-reinforcement effects between trigger inputs and trojan models\mcite{imc}.

\begin{mtbox}{\small Remark\,15}
    {\em \small There often exists universal, single-pixel perturbation with respect to trojan models (but not clean models).}
\end{mtbox}

{\bf RQ$\bm{_2}$:\,{\em Is it necessary to use regular-shaped triggers?}}

The triggers in the existing attacks are mostly regular-shaped (\textit{e.g.}\xspace,\,square), which seems a common design choice. We explore the impact of trigger shape on attack efficacy. We fix $|m|$\,=\,9 but select the positions of $|m|$ pixels independently and randomly. Table\mref{tab:scattered} compares {\em \small ASR}\xspace under the settings of regular and random triggers.

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c|c|c|c|c|c}
{\bf Trigger Setting} & {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Lb}\xspace & {\sc Imc}\xspace\\
\hline
\hline
Regular & 72.4 & 91.5 & 79.2 & 100.0 & 100.0 \\
Random & 97.6 & 98.5 & 92.7 & 97.6 & 94.5 \\
\end{tabular}
\caption{\footnotesize Comparison of regular and random triggers. \label{tab:scattered}}}
\end{table}

Except {\sc Lb}\xspace and {\sc Imc}\xspace which already attain extremely high {\em \small ASR}\xspace under the regular-trigger setting, all the other attacks achieve higher {\em \small ASR}\xspace under the random-trigger setting. For instance, the {\em \small ASR}\xspace of {\sc Bn}\xspace increases by 25.2\%. This may be explained by that lifting the spatial constraint on the trigger entails a larger optimization space for the attacks.

\begin{mtbox}{\small Remark\,16}
    {\em \small Lifting spatial constraints on trigger patterns tends to lead to more effective attacks.}
\end{mtbox}

{\bf RQ$\bm{_3}$:\,{\em Is the ``neuron-separation'' guidance effective?}}

A common search strategy for trigger patterns is using the neuron-separation guidance: searching for triggers that activate neurons rarely used by clean inputs\mcite{trojannn}. Here, we validate this guidance by measuring the {\em \small NSR}\xspace (\msec{sec:attack-metric}) of benign and trojan models before and after {\sc Fp}\xspace, as shown in Table\mref{tab:jac-index}.

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{3pt}
    \begin{tabular}{c|c|c|c|c|c|c|c}
           {\bf Fine-Pruning}      & -- & {\sc Bn}\xspace   & {\sc Tnn}\xspace  & {\sc Rb}\xspace  & {\sc Lb}\xspace   & {\sc Abe}\xspace  & {\sc Imc}\xspace  \\
        \hline
        \hline
        Before  & 0.03 & 0.59 & 0.61 & \cellcolor{Red}0.65 & 0.61 & 0.54 & 0.64 \\
        After & 0.03 & 0.20 & 0.19 & 0.27 & 0.37 & 0.18 & \cellcolor{Red}0.38 \\
    \end{tabular}
    \caption{\footnotesize {\em \small NSR}\xspace of benign and trojan models before and after {\sc Fp}\xspace. \label{tab:jac-index}}}
\end{table}

Across all the cases, compared with its benign counterpart, the trojan model tends to have higher {\em \small NSR}\xspace, while fine-tuning reduces {\em \small NSR}\xspace significantly. More effective attacks (\textit{cf}.\xspace Figure\mref{tab:attack_summary}) tend to have higher {\em \small NSR}\xspace (\textit{e.g.}\xspace, {\sc Imc}\xspace). We thus conclude that the neuron-separation heuristic is in general valid.

\begin{mtbox}{\small Remark\,17}
    {\em \small The separation between the neurons activated by clean and trigger inputs is an indicator of attack effectiveness.}
\end{mtbox}

\subsection{Attack -- Optimization}

We now examine the optimization strategies used by exiting attacks and explore potential improvement.

\vspace{1pt}
{\bf RQ$\bm{_4}$:\,{\em Is it necessary to start from benign models?}}

To forge a trojan model, a common strategy is to re-train a benign, pre-trained model. Here, we challenge this practice by evaluating whether re-training a benign model leads to more effective attacks than training a trojan model from scratch.

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c|c|c|c|c|c}
{\bf Training Strategy} & {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Lb}\xspace & {\sc Imc}\xspace \\
\hline
\hline
Benign model re-training & 72.4 & 91.5 & 79.2 & 100.0 & 100.0 \\
Training from scratch & 76.9 & 98.9 & 81.2 & 100.0  & 100.0 \\
\end{tabular}
\caption{\footnotesize {\em \small ASR}\xspace of trojan models by training from scratch and re-training from benign models. \label{tab:scratch}}}
\end{table}

Table\mref{tab:scratch} compares the {\em \small ASR}\xspace of trojan models generated using the two strategies. Except {\sc Lb}\xspace and {\sc Imc}\xspace achieving similar {\em \small ASR}\xspace in both settings, the other attacks observe marginal improvement if they are trained from scratch. For instance, the {\em \small ASR}\xspace of {\sc Tnn}\xspace improves by 7.4\%. One possible explanation is as follows. Let $f$ and $\ssup{f}{*}$ represent the benign and trojan models, respectively. In the parameter space, re-training constrains the search for $\ssup{f}{*}$ within in $f$'s vicinity, while training from scratch searches for $\ssup{f}{*}$ in the vicinity of a randomly initialized configuration, which may lead to better starting points.

\begin{mtbox}{\small Remark\,18}
    {\em \small Training from scratch tends to lead to more effective attacks than benign-model re-training.}
\end{mtbox}

{\bf RQ$\bm{_5}$:\, {\em Is it feasible to exploit model architectures?}}

Most attacks train trojan models in a model-agnostic manner, ignoring their unique architectures (\textit{e.g.}\xspace, residual block). We explore the possibility of exploiting such features.

We consider the skip-connection structures in many DNNs (\textit{e.g.}\xspace,\,ResNet) and attempt to improve the gradient backprop in training trojan models. In such networks, gradients propagate through both skip connections and residual blocks. By setting the weights of gradients from skip connections or residual blocks, it amplifies the gradient update towards inputs or model parameters\mcite{skip-connection}. Specifically, we modify the backprop procedure in {\sc Imc}\xspace by setting a decay coefficient $\gamma = 0.5$ for the gradient through skip connections, with {\em \small ASR}\xspace improvement over normal training shown in Figure\mref{fig:sgm}.

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/sgm.eps, width = 80mm}
    \caption{\footnotesize {\em \small ASR}\xspace improvement by reducing skip-connection gradients ($\alpha$\,=\,0.9). \label{fig:sgm}}
\end{figure}

Observe that by reducing the skip-connection gradients, it marginally improves the {\em \small ASR}\xspace of {\sc Imc}\xspace especially for small triggers (\textit{e.g.}\xspace, $|m|$\,=\,2$\times$2). We consider searching for the optimal $\gamma$ to maximize attack efficacy as our ongoing work.

\begin{mtbox}{\small Remark\,19}
    {\em \small It is feasible to exploit skip-connection structures to improve attack efficacy marginally.}
\end{mtbox}

{\bf RQ$\bm{_6}$:\,{\em How to mix clean and trigger inputs in training?}}

To balance attack efficacy and specificity, the adversary often mixes clean and trigger inputs in training trojan models. There are typically three mixing strategies: \mcounter{i} dataset-level -- mixing trigger inputs ${\mathcal{T}}_\mathrm{t}$ with clean inputs ${\mathcal{T}}_\mathrm{c}$ directly, \mcounter{ii} batch-level -- adding trigger inputs to each batch of clean inputs during training, and \mcounter{iii} loss-level -- computing and aggregating the average losses of ${\mathcal{T}}_\mathrm{t}$ and ${\mathcal{T}}_\mathrm{c}$. Here, we fix the mixing coefficient $\lambda$\,=\,0.01 and compare the effectiveness of different strategies.

\begin{table}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c|c|c|c|c|c}
{\bf Mixing Strategy} & {\sc Bn}\xspace & {\sc Tnn}\xspace & {\sc Rb}\xspace & {\sc Lb}\xspace & {\sc Imc}\xspace \\
\hline
\hline
Dataset-level & 59.3 & 72.2 & 46.2 & 99.6 & 92.0 \\
Batch-level & \cellcolor{Red}72.4 & \cellcolor{Red}91.5 & \cellcolor{Red}79.2 & \cellcolor{Red}100.0 & \cellcolor{Red}100.0 \\
Loss-level & 21.6 & 22.9 & 18.1 & 33.6 & 96.5  \\
\end{tabular}
\caption{\footnotesize Impact of mixing strategies on attack efficacy ($\alpha$\,=\,0.0, $\lambda$\,=\,0.01).\label{tab:mix}}}
\end{table}

We observe in Table\mref{tab:mix} that across all the cases, the batch-level mixing strategy leads to the highest {\em \small ASR}\xspace. This can be explained as follows. With dataset-level mixing, the ratio of trigger inputs in each batch tends to fluctuate significantly due to random shuffling, resulting in inferior training quality. With loss-level mixing,
$\lambda$\,=\,0.01 results in fairly small gradients of trigger inputs, equivalent to setting a overly small learning rate. In comparison, batch-level mixing fixes the ratio of trigger inputs in each batch and the weight of their gradients in updating trojan models.

\begin{mtbox}{\small Remark\,20}
    {\em \small Batch-level mixing tends to lead to the most effective training of trojan models.}
\end{mtbox}

{\bf RQ$\bm{_7}$:\,{\em How to optimize the trigger pattern?}}

An attack involves optimizing both the trigger pattern and the trojan model. The existing attacks use 3 typical strategies: \mcounter{i} Pre-defined trigger -- it fixes the trigger pattern and only optimizes the trojan model.
\mcounter{ii} Partially optimized trigger -- it optimizes the trigger pattern in a pre-processing stage and optimizes the trojan model. \mcounter{iii} Trigger-model co-optimization -- it optimizes the trigger pattern and the trojan model jointly during training. Here, we implement 3 variants of {\sc Bn}\xspace that use these optimization strategies, respectively. Figure\mref{fig:optim_term} compares their {\em \small ASR}\xspace under varying trigger transparency. Observe that the trigger-optimization strategy has a significant impact on {\em \small ASR}\xspace, especially under high transparency. For instance, if $\alpha$\,=\,0.9, the co-optimization strategy improves {\em \small ASR}\xspace by over 60\% from the non-optimization strategy.

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/optimization.eps, width = 75mm}
    \caption{\footnotesize Impact of trigger optimization. \label{fig:optim_term}}
\end{figure}

\begin{mtbox}{\small Remark\,21}
    {\em \small Optimizing the trigger pattern and the trojan model jointly leads to more effective attacks.}
\end{mtbox}

\subsection{Defense -- Interpretability}

{\bf RQ$\bm{_8}$:\,{\em Does interpretability help mitigate backdoor attacks?}}

The interpretability of DNNs explain how they make predictions for given inputs\mcite{selvaraju:gradcam,fong:mask}. Recent studies\mcite{Tao:nips:2018,Guo:2018:ccs} show that such interpretability helps defend against adversarial attacks. Here, we explore whether it mitigates backdoor attacks. Specifically, for a pair of benign-trojan models and 100 pairs of clean-trigger inputs, we generate the attribution map\mcite{selvaraju:gradcam} of each input with respect to both models and ground and target classes, with an example shown in Figure\mref{fig:example}.

\begin{figure}[!ht]
    \centering
    \epsfig{file = figures/examples.eps, width = 75mm}
    \caption{\footnotesize Sample attribution maps of clean and trigger inputs with respect to benign and trojan models ($\alpha$\,=\,0.0, {ImageNet}\xspace). \label{fig:example}}
\end{figure}

We measure the difference ($\ell_1$-norm normalized by image size) of attribution maps of clean and trigger inputs. Observe in Table\mref{tab:map-difference} that their attribution maps with respect to the target class differ significantly on the trojan model, indicating the possibility of using interpretability to detect the attack. Yet, it requires further study whether the adversary may adapt the attack to deceive such detection\mcite{adv}.

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}

    \begin{tabular}{c|c|c|c}
    \multicolumn{2}{c|}{\bf Benign model} &  \multicolumn{2}{c}{\bf Trojan model} \\
    \hline
    {\bf Original class} & {\bf Target class} &  {\bf Original class} & {\bf Target class} \\
    \hline
    \hline
    0.08\% & 0.12\% & 0.63\% &\cellcolor{Red}8.52\% \\
    \end{tabular}
    \caption{\footnotesize Heatmap difference of clean and trigger inputs ($\alpha$\,=\,0.0, {ImageNet}\xspace).  \label{tab:map-difference} }}
\end{table}

\begin{mtbox}{\small Remark\,22}
    {\em \small It seems feasible to exploit interpretability to defend against backdoor attacks.}
\end{mtbox}

\subsection{Defense -- Mitigation}

{\bf RQ$\bm{_{9}}$:\,{\em Is unlearning able to cleanse backdoors?}}

Once a backdoor is detected, the natural follow-up is to sanitize the trojan model. We explore whether the existing  methods remove the backdoors. We exemplify with unlearning\mcite{neural-cleanse}: it applies the recovered trigger on clean data to generate trigger inputs and uses such inputs together with their ground-truth classes to re-trains the trojan model.

\begin{figure*}[!ht]
    \centering
    \epsfig{file = figures/adaptation.eps, width = 150mm}
    \caption{\footnotesize Performance of non-adaptive and adaptive {\sc Imc}\xspace against representative defenses ($\alpha$\,=\,0.0). \label{fig:adaptation}}
\end{figure*}

\begin{table}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{3pt}
    \begin{tabular}{c|cc|cc|cc || c c }
        \multirow{2}{*}{\bf Class}  &   \multicolumn{2}{c|}{\bf Before} &  \multicolumn{2}{c|}{\bf After (detected)} &  \multicolumn{2}{c||}{\bf After (original)} & \multicolumn{2}{c}{\bf Adapted} \\
        \cline{2-9}
& {\em \small ASR}\xspace & {\em \small MLN}\xspace & {\em \small ASR}\xspace & {\em \small MLN}\xspace & {\em \small ASR}\xspace & {\em \small MLN}\xspace & {\em \small ASR}\xspace & {\em \small MLN}\xspace\\
        \hline
        \hline
    0 &    \cellcolor{Red}100.0 & \cellcolor{Red}17.75 &  \cellcolor{Red}94.96 & \cellcolor{Red}16.00  & \cellcolor{Red}94.76 & \cellcolor{Red}15.83 & \cellcolor{Blue}87.13 & \cellcolor{Blue}16.00    \\
    1  &  1.30 & 94.38 & \cellcolor{Red}94.39 & \cellcolor{Red}16.00 &  \cellcolor{Red}94.37 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.44 & \cellcolor{Blue}16.00 \\
    2  &  99.45 & 100.72 &  \cellcolor{Red}94.20 & \cellcolor{Red}16.00 & \cellcolor{Red}93.97 & \cellcolor{Red}16.00 &
    \cellcolor{Blue}86.64 & \cellcolor{Blue}16.00  \\
    3  &  59.52 & 101.14 & \cellcolor{Red}94.88 & \cellcolor{Red}16.00  & \cellcolor{Red}94.72 & \cellcolor{Red}15.83 &\cellcolor{Blue}87.10 & \cellcolor{Blue}16.00  \\
    4 &   0.79 & 100.57 & \cellcolor{Red}93.98 & \cellcolor{Red}16.00 &  \cellcolor{Red}94.39 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.58& \cellcolor{Blue}16.00\\
    5  &  6.12 & 20.86 &  \cellcolor{Red}94.42 & \cellcolor{Red}16.00 & \cellcolor{Red}94.51 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.68 & \cellcolor{Blue}16.00 \\
    6  &  29.77 & 100.21 & \cellcolor{Red}94.35 & \cellcolor{Red}16.00  & \cellcolor{Red}94.33 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.66 & \cellcolor{Blue}16.00 \\
    7 & 6.07 & 100.57 &  \cellcolor{Red}94.48 & \cellcolor{Red}16.00 & \cellcolor{Red}94.36 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.56 & \cellcolor{Blue}16.00 \\
    8 &  27.37 & 101.53 &  \cellcolor{Red}94.43 & \cellcolor{Red}16.00 & \cellcolor{Red}94.16 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.69 & \cellcolor{Blue}16.00 \\
    9  & 99.77 & 100.47 & \cellcolor{Red}94.35 & \cellcolor{Red}16.00 & \cellcolor{Red}94.19 & \cellcolor{Red}16.00 & \cellcolor{Blue}86.41 & \cellcolor{Blue}16.00\\
    \end{tabular}
    \caption{\footnotesize Impact of unlearning and adaptation to {\sc Strip}\xspace (detected -- unlearning using trigger detected by {\sc Nc}\xspace; original -- unlearning using ground-truth trigger; adapted -- adaptation to {\sc Strip}\xspace). \label{tab:unlearning}}}
\end{table}

We apply unlearning with \mcounter{i} the original trigger and \mcounter{ii} the trigger recovered by {\sc Nc}\xspace. In both cases, according to {\sc Nc}\xspace, the unlearning successfully cleanses all the backdoors (details in Appendix\mref{sec:additional}). We then apply {\sc Abs}\xspace to re-inspect the unlearned models and measure the {\em \small ASR}\xspace and {\em \small MLN}\xspace of each class before and after unlearning,
with results listed in Table\mref{tab:unlearning}. Interestingly, before unlearning, only class 0 is infected with a backdoor; after unlearning in both cases, all the classes are infected with backdoors (with {\em \small ASR}\xspace above 90\% and {\em \small MLN}\xspace below 16)!

\begin{mtbox}{\small Remark\,23}
    {\em \small Unlearning tends to introduce new vulnerabilities.}
\end{mtbox}

\subsection{Defense -- Evadability}

{\bf RQ$\bm{_{10}}$:\,{\em Are the existing defenses evadable?}}

We now explore whether the existing defenses are potentially evadable by adaptive attacks. We select {\sc Imc}\xspace as the basic attack, due to its flexible optimization framework, and consider {\sc Mp}\xspace, {\sc Ar}\xspace, {\sc Strip}\xspace, and {\sc Abs}\xspace as the representative defenses from the categories in Table\mref{tab:defense_summary}. Specifically, we adapt {\sc Imc}\xspace to each defense (details deferred to Appendix\mref{sec:details}).

We compare the efficacy of non-adaptive and adaptive {\sc Imc}\xspace, as shown in Figure\mref{fig:adaptation}. Observe that across all the cases, the adaptive {\sc Imc}\xspace significantly outperforms the non-adaptive one. For instance, under $|m|$\,=\,6$\times$6, it increases the {\em \small ASR}\xspace with respect to {\sc Mp}\xspace by 80\% and reduces the {\em \small TPR}\xspace of {\sc Strip}\xspace by over 0.85. Also note that a larger trigger size leads to more effective adaptive attack, as it entails a larger optimization space.

\begin{mtbox}{\small Remark\,24}
    {\em \small Most existing defenses are potentially evadable by adaptive attacks.}
\end{mtbox}

{\bf RQ$\bm{_{11}}$:\,{\em Are ensemble defenses more robust?}}

Given that different defenses focus on distinct aspects of backdoors, a promising approach is to integrate multiple, complementary defenses. Here, we exemplify with {\sc Strip}\xspace, which exploits trigger pattern anomaly, and {\sc Abs}\xspace, which focuses on neuron activation anomaly. We apply {\sc Abs}\xspace to inspect the trojan model generated by {\sc Imc}\xspace adapted to {\sc Strip}\xspace, with results in Table\mref{tab:unlearning}. Observe that {\sc Abs}\xspace detects all the classes infected with backdoors. We have the following explanation. Adapting to {\sc Strip}\xspace requires to train the trojan model using trigger inputs of high transparency together with their original classes (\textit{cf}.\xspace\,Appendix\mref{sec:details}). This is essentially unlearning to forget high-transparency triggers while keeping low-transparency triggers effective. Thus, the adaptation to {\sc Strip}\xspace tends to make the trojan model more susceptible to {\sc Abs}\xspace.

\begin{mtbox}{\small Remark\,25}
    {\em \small Integrating defenses against trigger inputs and trojan models may lead to robustness against adaptive attacks.}
\end{mtbox}

\section{Introduction}

Today's deep learning (DL) systems are large, complex software artifacts. With the increasing system complexity and training cost, it becomes not only tempting but also necessary to exploit pre-trained deep neural networks (DNNs) in building DL systems. It was estimated that as of 2016, over 13.7\% of DL-related repositories on GitHub re-use at least one pre-trained DNN\mcite{Ji:2018:ccsa}. On the upside, this ``plug-and-play'' paradigm greatly simplifies the development cycles\mcite{Sculley:2015:nips}. On the downside, as most pre-trained DNNs are contributed by untrusted third parties\mcite{modelzoo}, their lack of standardization or regulation entails profound security implications.

In particular, pre-trained DNNs can be exploited to launch {\em neural backdoor} attacks\mcite{badnet,trojannn,imc}, one immense threat to the security of DL systems. In such attacks, a maliciously crafted DNN (``trojan model'') forces its host system to misbehave once certain pre-defined conditions (``triggers'') are present but functions normally otherwise. Such attacks can result in consequential damages such as misleading autonomous vehicles to crashing\mcite{Versprille:2015:news}, maneuvering video surveillance to miss illegal activities\mcite{Cooper:2014:news}, and manipulating biometric authentication to allow improper access\mcite{Biggio:2012:spr}.

Motivated by this, intensive research has been conducted on neural backdoors, leading to a plethora of attacks that craft trojan model via exploiting various properties (\textit{e.g.}\xspace,\,neural activation patterns)\mcite{badnet,targeted-backdoor,trojannn,invisible-backdoor,Suciu:2018:sec,latent-backdoor} and defenses that mitigate trojan models during  inspection\mcite{neural-cleanse,abs,deep-inspect,tabor,neuron-inspect,fine-pruning} or detect trigger inputs at inference\mcite{strip,active-clustering,randomized-smoothing,neo}. With the rapid development of new attacks/defenses, a number of open questions have emerged:

\vspace{2pt}
RQ$_1$ -- {\em How effective, evasive, and transferable are the existing attacks?}

\vspace{1pt}
RQ$_2$ -- {\em How robust, utility-preserving, and generic are the existing defenses?}

\vspace{1pt}
RQ$_3$ -- {\em How do various factors (\textit{e.g.}\xspace, model architectures) impact the performance of different attacks/defenses?}

\vspace{1pt}
RQ$_4$ -- {\em What are the best practices (\textit{e.g.}\xspace, optimization strategies) to operate different attacks/defenses?}

\vspace{1pt}
RQ$_5$ -- {\em How can the existing backdoor attacks/defenses be further improved?}

\vspace{2pt}

Despite their importance for assessing and mitigating the vulnerabilities incurred by pre-trained DNNs, these questions are largely unexplored due to the following challenges.

\vspace{1pt}
\underline{\em Non-holistic evaluations} -- Most studies conduct evaluations with a limited set of attacks/defenses, resulting in incomplete comparison. For instance, it is unknown whether the {{\sc Strip}\xspace} defense\mcite{strip} is effective against the newer {{\sc Abe}\xspace} attack\mcite{adv-backdoor}. Further, the evaluations often use simple, macro-level metrics, failing to comprehensively characterize given attacks/defenses. For instance, most studies use attack success rate ({\em \small ASR}\xspace) and clean accuracy drop ({\em \small CAD}\xspace) to assess an attack's performance, yet insufficient to describe the attack's ability of trading off between the two metrics.

\vspace{1pt}
\underline{\em Non-unified platforms} -- Due to the lack of unified benchmarks, different attacks/defenses are often evaluated under varying configurations, leading to non-comparable conclusions. For instance, {{\sc Tnn}\xspace}\mcite{trojannn} and {{\sc Lb}\xspace}\mcite{latent-backdoor} are evaluated with distinct trigger definitions (\textit{i.e.}\xspace,\,shape, size, and transparency), datasets, and DNNs, making it difficult to directly compare their effectiveness and evasiveness.

\vspace{1pt}
\underline{\em Non-adaptive attacks} --  The evaluations of existing defenses (\textit{e.g.}\xspace,\,\mcite{fine-pruning,neural-cleanse,strip,tabor}) often assume static, non-adaptive attacks, without fully accounting for the adversary's possible countermeasures, which however is critical for modeling the adversary's optimal strategies and assessing the attack vulnerabilities in realistic settings.

\subsection*{Our Work} To this end, we design, implement, and evaluate {\sc \small TrojanZoo}\xspace, an open-source platform for assessing neural backdoor attacks/defenses in a holistic, unified, and practical manner. Our contributions are summarized as follows.

\vspace{1pt}
{\bf Platform --} To our best knowledge, {\sc \small TrojanZoo}\xspace represents the first open-source platform designed for evaluating neural backdoor attacks/defenses. To date, {\sc \small TrojanZoo}\xspace has incorporated 12 representative attacks, 15 state-of-the-art defenses, 6 attack performance metrics, 10 defense utility metrics, as well as a benchmark suite of 5 DNN models, 5 downstream models, and 6 datasets. All the modules (\textit{e.g.}\xspace,\,attacks, defenses, and models) are provided as Dockerfiles for easy installation and deployment. Further, {\sc \small TrojanZoo}\xspace implements utility tools for in-depth analysis of attack-defense interactions, including measuring feature-space similarity, tracing neural activation patterns, and comparing attribution maps.

\vspace{1pt}
{\bf Assessment --} Leveraging {\sc \small TrojanZoo}\xspace, we conduct a systematic study of existing attacks/defenses. The attacks are evaluated in terms of effectiveness, evasiveness, and transferability, while the defenses are assessed in terms of robustness, utility-preservation, and genericity. We make a number of interesting observations:
\mcounter{i} most attacks manifest strong ``mutual-reinforcement'' effects in the effectiveness-evasiveness trade-off;
\mcounter{ii} DNN architectures that enable better feature extraction may also allow more effective propagation of trigger patterns; \mcounter{iii} more effective attacks (\textit{e.g.}\xspace,\,higher {\em \small ASR}\xspace) are also more likely to be detected by input filtering (\textit{e.g.}\xspace,\,\mcite{strip}); \mcounter{iv} weaker attacks (\textit{i.e.}\xspace,\,lower {\em \small ASR}\xspace) demonstrate higher transferability than stronger ones; \mcounter{v} model-inspection defenses (\textit{e.g.}\xspace,\,\mcite{abs}) often uncover backdoors non-identical to, but overlapping with, the ones injected by the adversary. Our evaluation unveils the strengths and limitations of existing attacks/defenses as well as their intricate interactions.

\vspace{1pt}
{\bf Exploration --} We further explore improving existing attacks/defenses, leading to a set of previously unknown findings such as \mcounter{i} one-pixel triggers suffice (over 95\% {\em \small ASR}\xspace) for many attacks; \mcounter{ii} training from scratch seems more effective than re-training benign models to forge trojan models; \mcounter{iii} leveraging DNN architectures (\textit{e.g.}\xspace,\,skip connection) in optimizing trojan models marginally improves attack effectiveness; \mcounter{iv} optimizing trigger patterns along with trojan models improves both attack effectiveness and evasiveness; \mcounter{iv} sanitizing trojan models via unlearning\mcite{neural-cleanse}, while fixing existing backdoors, may introduce new vulnerabilities; \mcounter{v} while most defenses are vulnerable to adaptive attacks, integrating complementary ones (\textit{e.g.}\xspace,\,model inspection and input filtering) significantly enhances defense robustness. We envision that our findings will facilitate future research on neural backdoors and shed light on designing and building DL systems in a more secure and informative manner.\footnote{All the data, models, and code of the paper are open-sourced at \url{https://github.com/ain-soph/trojanzoo}}

\section{Attacks, Defenses, and Metrics}

In its current implementation, {\sc \small TrojanZoo}\xspace incorporates 9 representative neural backdoor attacks and 10 state-of-the-art backdoor mitigation methods, as well as 6 attack performance metrics and 4 defense utility metrics, as detailed below.

\subsection{Attacks}

While neural backdoor attacks can be characterized from a range of aspects, here we focus on 5 key design choices by the adversary that directly impact attack performance.

\begin{mitemize}
\item {\em Architecture modifiability} -- whether the attack is allowed to change the benign DNN architecture. Being able to modify both the model architecture and parameters enables the adversary with a larger attack spectrum, but also renders the trojan model more susceptible to certain defenses (e.g., model specification checking).

\item {\em Trigger optimizability} -- whether the attack uses a fixed, pre-defined trigger or optimizes the trigger during crafting the trojan model. Being able to optimize the trigger often leads to stronger attacks with respect to given desiderata (e.g., trigger stealthiness).

\item {\em Poisoning data detectability} -- whether the attack optimizes the detectability of poisoning data. Under the setting that the user controls the model training while the adversary influences the training through injecting the poisoning data (e.g., transfer learning), it is desirable that the poisoning data is indistinguishable from clean data.

\item {\em Fine-tuning survivability} -- whether the attack retains its effectiveness if the model is fine-tuned in the downstream task. A pre-trained (trojan) model is often composed with a classifier/regressor and fine-tuned using the data from the downstream task. It is desirable to ensure that the backdoor remains effective after the fine-tuning operation.

\item {\em Defense adaptivity} -- whether the attack is optimizable to evade possible defenses. To ensure the attack effectiveness, it is essesntial to maximize the evasiveness of trojan model and trigger inputs with respect to the deployed defense mechanisms.
\end{mitemize}

Table\mref{tab:attack_summary} summarizes the representative neural backdoor attacks currently implemented in {\sc \small TrojanZoo}\xspace, which are characterized along the above 5 key factors.

\vspace{1pt}
{\bf BadNet} ({\sc Bn}\xspace)\mcite{badnet}, as the simplest neural backdoor attack, uses a pre-defined trigger $r$ (including its shape, position, and pattern), generates a set of trigger poisoning inputs $\{ (x\oplus r, t)\}$, and forges the trojan model $f$ by re-training a benign DNN $f$ with such poisoning data.

\vspace{1pt}
{\bf TrojanNN} ({\sc Tnn}\xspace)\mcite{trojannn}, compared with {\sc Bn}\xspace, partially optimizes the trigger $r$. It pre-defines $r$ in terms of its shape and position, optimizes its pattern to activate a subset of neurons rarely activated by clean inputs in a pre-processing step, and then forges $f$ by re-training $f$ in a manner similar to {\sc Bn}\xspace.

\vspace{1pt}
{\bf Reflection Backdoor} ({\sc Rb}\xspace)\mcite{reflection-backdoor} optimizes trigger stealthiness by defining $r$ as the physical reflection of a clean image: $r = x_r \otimes k$, where $x_r$ is a reflection image, $k$ is a convolution kernel, and $\otimes$ is the convolution operator. To this end, {\sc Rb}\xspace selects $x_r$ from a candidate pool and forges $f$ by re-training $f$ using inputs embedded with $r$ in a manner similar to {\sc Bn}\xspace.

\vspace{1pt}
{\bf Targeted Backdoor} ({\sc Tb}\xspace)\mcite{targeted-backdoor} also optimizes trigger stealthiness. It partially defines $r$ in terms of its shape and pattern but randomly generates its position, which makes $f$ effective regardless of $r$'s position inside $x$ and allows the adversary to maximizes $r$'s evasiveness by placing it at the most plausible position (e.g., an eyewear watermark over eyes).

\vspace{1pt}
{\bf Latent Backdoor} ({\sc Lb}\xspace)\mcite{latent-backdoor} defines $r$ in a manner similar to {\sc Tnn}\xspace, but accounts for the impact of downstream fine-tuning by optimizing $f$ with respect to the latent representations rather than final predictions. Specifically, it instantiates \meq{eq:opt} with the following loss function:
\begin{align}
\ell(f(x \oplus r), t) = \Delta(f(x \oplus r), \phi_t)
\end{align}
where $\Delta$ measures the difference of the latent representations of two inputs and $\phi_t$ is the representation of class $t$, which features the minimum distance to all the inputs in $t$ (i.e., $\phi_t = \min_{\phi} {\mathbb{E}}_{(x, t) \in \mathcal{T}} [ \Delta ( f(x), \phi_t) ] $).

\vspace{1pt}
{\bf Embarrassingly Simple Backdoor} ({\sc Esb}\xspace)\mcite{embarassingly-simple-backdoor}, different from other attacks that forge $f$ by re-training $f$, directly modifies $f$'s architecture by adding a trojan module which recognizes $r$ and overwrites the prediction as $t$ if $r$ is detected. Without perturbing $f$'s original configuration, $f$ retains $f$'s predictive power regarding clean inputs. \ting{shall we consider the ``poisoning data detectability'' not applicable?}

\vspace{1pt}
{\bf Clean-Label Backdoor} ({\sc Clb}\xspace)\mcite{clean-label-backdoor} assumes the transfer learning setting in which the adversary forges $f$ by indirectly polluting the training data ${\mathcal{T}}$. To evade possible data filtering, {\sc Clb}\xspace generates (via either adversarial perturbation or adversarial generative networks) stealthy poisoning inputs that appear to be consistent with their labels and thus seem clean even upon human inspection.

\vspace{1pt}
{\bf Adversarial Backdoor Embedding} ({\sc Abe}\xspace)\mcite{adv-backdoor} accounts for possible defenses in forging $f$. In optimizing \meq{eq:opt}, {\sc Abe}\xspace also optimizes the indistinguishability of the latent representations of trigger and clean inputs. Specifically, it uses a discriminative network $d$ that predicts the representation of a given input $x$ as trigger or clean. Formally, the loss function is defined as $\Delta(d\circ f(x), b(x))$, where $b(x)$ encodes whether $x$ is trigger or clean, while $f$ and $d$ are trained using an adversarial learning framework\mcite{Goodfellow:2014:nips}.

\vspace{1pt}
{\bf Input-Model Co-optimization} ({\sc Imc}\xspace)\mcite{imc} is motivated by the ``mutual reinforcement'' effect between $r$ and $f$: optimizing one properly may significantly amplify the effectiveness of the other. Different from other attacks that solve \meq{eq:opt} by first (partially) pre-defining $r$ and then optimizing $f$, {\sc Imc}\xspace optimizes $r$ and $f$ simultaneously using a co-optimization framework, which significantly enlarges the search space of $r$ and $f$, leading to attacks that satisfy multiple desiderata (e.g., fine-tuning survivability and defense adaptivity).

\subsection{Attack Performance Metrics}

In its current implementation, {\sc \small TrojanZoo}\xspace incorporates 8 performance metrics to compare the effectiveness and evasiveness of different attacks.

\vspace{1pt}
{\bf Attack Success Rate} ({\em \small ASR}\xspace), which measures the likelihood that the system classifies trigger inputs to the target class $t$ designated by the adversary:
\begin{align}
\textrm{\small Attack Success Rate ({\em \small ASR}\xspace)} = \frac{\textrm{\small \# successful trials}}{\textrm{\small \# total trials}}
\end{align}
Typically higher {\em \small ASR}\xspace indicates more effective attacks.

\vspace{1pt}
{\bf Trojan Misclassification Confidence} ({\em \small TMC}\xspace), which is the average confidence score assigned to class $t$ by the system with respect to the trigger inputs in successful attacks. Intuitively, {\em \small TMC}\xspace complements {\em \small ASR}\xspace and measures attack efficacy from another perspective.

\vspace{1pt}
{\bf Clean Accuracy Drop} ({\em \small CAD}\xspace), which measures the difference of the classification accuracy of two systems built upon the benign model and its trojan counterpart with respect to clean inputs; {\em \small CAD}\xspace measures the attack specificity ({\em cf.} Figure\mref{fig:triangle}), that is, whether the attack directs its influence to trigger inputs only.

\vspace{1pt}
{\bf Clean Classification Confidence} ({\em \small CCC}\xspace), which measures the average confidence score assigned to their ground-truth classes by the system with respect to clean inputs; {\em \small CCC}\xspace complements {\em \small CAD}\xspace by measuring attack specificity from the perspective of classification confidence.

\vspace{1pt}
{\bf Efficacy-Specificity AUC} ({\em \small AUC}\xspace), which quantifies the aggregated trade-off between attack efficacy (measured by {\em \small ASR}\xspace) and attack specificity (measured by {\em \small CAD}\xspace). As revealed in\mcite{imc}, there exists an intricate balance: at a proper cost of specificity, it is possible to significantly improve efficacy, and vice versa; {\em \small AUC}\xspace measures the area under the {\em \small ASR}\xspace-{\em \small CAD}\xspace curve. Intuitively, smaller {\em \small AUC}\xspace implies more significant trade-off effect.

\subsection{Defenses}

\begin{table*}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{1pt}
\begin{tabular}{r|c|c|c|c|c|c|l}
        \multirow{2}{*}{\bf Neural Backdoor Defense} & \multirow{2}{*}{\bf Category}  & \multicolumn{2}{c|}{\bf Mitigation} &  \multicolumn{3}{c|}{\bf Detection Target} & \multirow{2}{*}{\bf Design Rationale} \\
        \cline{3-7}
            & &  {\bf Input} & {\bf Model} & {\bf Input} & {\bf Model} &  {\bf Trigger} &  \\

    \hline
    \hline

    Randomized-Smoothing ({\sc Rs}\xspace)\mcite{randomized-smoothing} & \multirow{3}{*}{\minitab[c]{Input\\Transformation}} &  $\checkmark$ & & & & &   ${\mathcal{A}}$'s fidelity ($x$'s and $x$'s surrounding class boundaries)  \\
    Downsampling-Upsampling ({\sc Du}\xspace)  &  &  $\checkmark$  & &  & & & ${\mathcal{A}}$'s fidelity ($x$'s and $x$'s high-level features) \\
    Compressing-Decompressing ({\sc Cd}\xspace)\mcite{magnet}  &  &  $\checkmark$  & &  & & & ${\mathcal{A}}$'s fidelity ($x$'s and $x$'s manifold projections) \\

    \hline

    Activation-Clustering ({\sc Ac}\xspace)\mcite{active-clustering} & Input & & & $\checkmark$   & & & distinct activation patterns of $\{x\}$ and $\{x\}$ \\
    STRIP ({\sc Strip}\xspace)\mcite{strip} & Filtering & & & $\checkmark$ & & & distinct entropy of $\{x\}$'s and $\{x\}$'s mixtures with clean inputs  \\

    \hline

    Adversarial-Retraining ({\sc Ar}\xspace)\mcite{madry:iclr:2018}  & Model & &  $\checkmark$  & & & & ${\mathcal{A}}$'s fidelity ($x$'s and $x$'s surrounding class boundaries)   \\
    Fine-Pruning ({\sc Fp}\xspace)\mcite{fine-pruning} & Sanitization & &  $\checkmark$  & & & & ${\mathcal{A}}$'s use of neurons rarely activated by clean inputs \\

    \hline

    NeuralCleanse ({\sc Nc}\xspace)\mcite{neural-cleanse} &  \multirow{5}{*}{\minitab[c]{Model\\Inpsection}}  & & & & $\checkmark$ & $\checkmark$ &  abnormally small perturbation from other classes to $t$ in $f$\\
    DeepInspect ({\sc Di}\xspace)\mcite{deep-inspect}  & & & & & $\checkmark$ & $\checkmark$ &  abnormally small perturbation from other classes to $t$ in $f$\\
    TABOR ({\sc Tabor}\xspace)\mcite{tabor}  & & & & & $\checkmark$ & $\checkmark$ & abnormally small perturbation from other classes to $t$ in $f$ \\
    NeuronInspect ({\sc Ni}\xspace)\mcite{neuron-inspect} & & & & & $\checkmark$ & &  distinct explanations of $f$ and $f$ with respect to clean inputs \\
    ABS ({\sc Abs}\xspace)\mcite{abs} & & & & &$\checkmark$ &$\checkmark$ & ${\mathcal{A}}$'s use of neurons elevating $t$'s prediction regardless of inputs  \\

\end{tabular}
\caption{Summary of representative neural backdoor defenses currently implemented in {\sc \small TrojanZoo}\xspace (${\mathcal{A}}$ -- backdoor attack, $x$ -- clean input, $x$ -- trigger input, $f$ -- benign model, $f$ -- trojan model, $t$ -- target class)  \label{tab:defense_summary}}}
\end{table*}

The existing defenses against neural backdoor attacks, according to their strategies, can be categorized as:

\begin{mitemize}
\item Model inspection -- which detects trojan models and/or recovers potential triggers at the model checking stage.
\item Model sanitization -- which sanitizes given models to mitigate the influence of potential backdoors yet without explicitly detecting trojan models.
\item Input filtering -- which detects trigger inputs at the inference stage.
\item Input transformation -- which reforms given inputs to mitigate the influence of potential triggers yet without explicitly detecting trigger inputs.

\end{mitemize}

Note that here we assume the setting of transfer learning or outsourced training, which precludes certain other defenses such as purging poisoning training data and re-training suspicious models\mcite{certified-purging,spectral-signature}. Table\mref{tab:defense_summary} summarizes the 13 representative neural backdoor defenses currently implemented in {\sc \small TrojanZoo}\xspace, which are detailed below.

The input sanitization methods exploit the high fidelity of backdoor attacks, that is, the attack ${\mathcal{A}}$ tend to retains the perceptual similarity of a clean input $x$ and its trigger counterpart $x$.

\vspace{1pt}
{\bf Randomized-Smoothing} ({\sc Rs}\xspace)\mcite{randomized-smoothing} exploits the premise that ${\mathcal{A}}$ retains the similarity of $x$ and $x$ in terms of their surrounding class boundaries, and classifies $x$ by averaging the predictions of the inputs within $x$'s vicinity (via adding Gaussian noise).

\vspace{1pt}
{\bf Downsampling-Upsampling} ({\sc Du}\xspace) exploits the premise that ${\mathcal{A}}$ retains the similarity of $x$ and $x$ in terms of their high-level features while the embedded trigger $r$ is typically not perturbation-tolerant. By first downsampling and then upsampling $x$, it is possible to mitigate $r$'s influence.

\vspace{1pt}
{\bf Compressing-Decompressing} ({\sc Cd}\xspace)\mcite{magnet} exploits the premise that ${\mathcal{A}}$ retains the similarity of $x$ and $x$ in terms of their projections to the manifold of clean data. To this end, we train an autoencoder to learn an approximate manifold and feed $x$ to the autoencoder, which projects $x$ to the manifold and then reconstructs an input potentially similar to $x$.

The input inspection methods attempt to distinguish clean and trigger inputs by leveraging their distinct characteristics.

\vspace{1pt}
{\bf Activation-Clustering} ({\sc Ac}\xspace)\mcite{active-clustering} distinguishes clean and trigger inputs by clustering their latent representations. Note that while {\sc Ac}\xspace is also applicable for purging poisoning training data, we consider its use as an input inspection method at inference time.

\vspace{1pt}
{\bf STRIP} ({\sc Strip}\xspace)\mcite{strip} mixes a given input up with a clean input and computes the self-entropy of the prediction of this mixture. If the input is embedded with a trigger, the mixture is still dominated by the trigger and tends to be misclassified to the target class, resulting in relatively low self-entropy.

The model sanitization methods aim to cleanse potential backdoors embedded in given models without explicitly detecting trojan models

\vspace{1pt}
{\bf Adversarial-Retraining} ({\sc Ar}\xspace)\mcite{madry:iclr:2018} applies adversarial training over the pre-trained model. At a high level, by considering trigger inputs as one special kind of adversarial inputs, {\sc Ar}\xspace improves the model robustness to backdoor attacks by im

\subsection{Defense Utility Metrics}

In its current implementation, {\sc \small TrojanZoo}\xspace incorporates 9 metrics to evaluate the robustness, utility-preservation, and genericity of various defenses. For each category of defenses, we define the metrics tailored to its objectives (\textit{e.g.}\xspace, detecting trigger inputs versus sanitizing trojan models).

For an input transformation method or an model sanitization method, we measure its effectiveness with respect to given attacks using the metrics of attack success rate ({\em \small ASR}\xspace) and trojan misclassification confidence ({\em \small TMC}\xspace), and its impact on clean inputs using the metrics of clean accuracy drop ({\em \small CAD}\xspace) and clean classification confidence ({\em \small CCC}\xspace), both compared with the case without the protection of this defense.

For an input filtering method, we measure its effectiveness using the metrics of true positive rate ({\em \small TPR}\xspace) and false positive rate ({\em \small FPR}\xspace) of correctly distinguishing clean and trigger inputs.

For an model inspection method, \mcounter{i} we evaluate its true positive rate ({\em \small TPR}\xspace) and false positive rate ({\em \small FPR}\xspace) of distinguishing trojan and clean models; \mcounter{ii} if it correctly identifies a trojan model, we further evaluate its accuracy ({\em \small ACC}\xspace) of identifying the target classes; \mcounter{iii} if both \mcounter{i} and \mcounter{ii} are correct, we further evaluate the quality of the recovered trigger, in terms of the $L_1$-norm ({\em \small L}$_1$\xspace) of the recovered trigger and the Jaccard index ({\em \small JAC}\xspace) of the recovered trigger and the original trigger:
\begin{equation}
    \textrm{\small Jaccard Index ({\em \small JAC}\xspace)} = \frac{|\textrm{\small recovered trigger} \cap \textrm{\small original trigger}|}{ |\textrm{\small recovered trigger} \cup \textrm{\small original trigger} |}
\end{equation}
which measures their intersection.

\section{0pt}{6pt}{4pt}

\usepackage[first=0,last=9]{lcg}
\usepackage{colortbl}
\definecolor{Gray}{gray}{0.8}
\colorlet{Red}{red!10!white}
\colorlet{Blue}{blue!10!white}

\usepackage{hyperref}
\renewcommand{\sectionautorefname}{\S}
\newcommand{\msec}[1]{\S\,\ref{#1}}
\newcommand{\mref}[1]{\,\ref{#1}}
\newcommand{\meq}[1]{Eq.\,\ref{#1}}
\newcommand{\mcite}[1]{\,\cite{#1}}
\newcommand{\textit{et al.}\xspace}{\textit{et al.}\xspace}
\newcommand{\textit{e.g.}\xspace}{\textit{e.g.}\xspace}
\newcommand{\textit{i.e.}\xspace}{\textit{i.e.}\xspace}
\newcommand{\textit{cf}.\xspace}{\textit{cf}.\xspace}
\newcommand{\mcounter}[1]{(\textit{#1})}

\newtcolorbox{mtbox}[1]{left=0.25mm, right=0.25mm, top=0.25mm, bottom=0.25mm, sharp corners, colframe=red!50!black, boxrule=0.5pt, title={#1}, fonttitle=\bfseries, coltitle=red!50!black, attach title to upper={\ --\ }}

\DeclareRobustCommand{\stirling}{\genfrac\{\}{0pt}{}}
\newcommand{\com}[2]{C_{#1}^{#2}}

\usepackage{scalerel}[2016/12/29]

\makeatletter
\providecommand{\leadsfrom}{%
  \mathrel{\mathpalette\reflect@squig\relax}%
}
\newcommand{\reflect@squig}[2]{%
  \reflectbox{$\m@th#1\leadsto$}%
}
\makeatother

\newcommand{\ssub{m}{\circ}}{\ssub{m}{\circ}}

\newcommand{\ssub{m}{*}}{\ssub{m}{*}}

\newcommand{x}{x}
\newcommand{x}{x}
\newcommand{\ssup{x}{*}}{\ssup{x}{*}}
\newcommand{\ssub{c}{\circ}}{\ssub{c}{\circ}}
\newcommand{t}{t}
\newcommand{\ssub{f}{b}}{\ssub{f}{b}}
\newcommand{\ssup{g}{*}}{\ssup{g}{*}}
\newcommand{f}{f}
\newcommand{\ssup{f}{*}}{\ssup{f}{*}}

\newcommand{{\small {\sc Grad}}\xspace}{{\small {\sc Grad}}\xspace}
\newcommand{{\small{\sc Cam}}\xspace}{{\small{\sc Cam}}\xspace}
\newcommand{{\small{\sc GradCam}}\xspace}{{\small{\sc GradCam}}\xspace}
\newcommand{{\small{\sc Mask}}\xspace}{{\small{\sc Mask}}\xspace}
\newcommand{{\small{\sc Rts}}\xspace}{{\small{\sc Rts}}\xspace}
\newcommand{{\small{\sc Ig}}\xspace}{{\small{\sc Ig}}\xspace}

\newcommand{{\small IDLS}\xspace}{{\small IDLS}\xspace}
\newcommand{{\small\texttt{AID}}\xspace}{{\small\texttt{AID}}\xspace}

\newcommand{{\small\texttt{PGD}}\xspace}{{\small\texttt{PGD}}\xspace}
\newcommand{{\small\texttt{StAdv}}\xspace}{{\small\texttt{StAdv}}\xspace}
\newcommand{{\small\texttt{C\&W}}\xspace}{{\small\texttt{C\&W}}\xspace}

\newcommand{{\small{\sc Fs}}\xspace}{{\small{\sc Fs}}\xspace}

\newcommand{DNN\xspace}{DNN\xspace}
\newcommand{DNNs\xspace}{DNNs\xspace}
\newcommand{ML\xspace}{ML\xspace}
\newcommand{DL\xspace}{DL\xspace}

\newcommand{{CIFAR10}\xspace}{{CIFAR10}\xspace}
\newcommand{{CIFAR100}\xspace}{{CIFAR100}\xspace}
\newcommand{{VGGFace2}\xspace}{{VGGFace2}\xspace}
\newcommand{{GTSRB}\xspace}{{GTSRB}\xspace}
\newcommand{{ImageNet}\xspace}{{ImageNet}\xspace}

\newcommand{{\small {\sf Word2Vec}}\xspace}{{\small {\sf Word2Vec}}\xspace}
\newcommand{{\small {\sf GloVe}}\xspace}{{\small {\sf GloVe}}\xspace}
\newcommand{{\small {\sf GoogLeNet}}\xspace}{{\small {\sf GoogLeNet}}\xspace}
\newcommand{{\small {\sf AlexNet}}\xspace}{{\small {\sf AlexNet}}\xspace}
\newcommand{{\small {\sf Inception.v3}}\xspace}{{\small {\sf Inception.v3}}\xspace}
\newcommand{{\small {\sf VGG}}\xspace}{{\small {\sf VGG}}\xspace}
\newcommand{{\small {\sf ResNet}}\xspace}{{\small {\sf ResNet}}\xspace}
\newcommand{{\small {\sf DenseNet}}\xspace}{{\small {\sf DenseNet}}\xspace}

\newcommand{\order}[1]{{#1}^{\textrm{th}}}

\newcommand\eqover{\mathrel{\overset{\makebox[0pt]{\mbox{\normalfont\tiny\sffamily def}}}{=}}}

\newcommand{{\em (Left)}}{{\em (Left)}}
\newcommand{{\em (Center)}}{{\em (Center)}}
\newcommand{{\em (Right)}}{{\em (Right)}}
\newcommand{{\em (Top)}}{{\em (Top)}}
\newcommand{{\em (Bottom)}}{{\em (Bottom)}}
\newcommand{{\em (a)}}{{\em (a)}}
\newcommand{{\em (b)}}{{\em (b)}}
\newcommand{{\em (c)}}{{\em (c)}}
\newcommand{{\em (d)}}{{\em (d)}}

\newcommand{\newterm}[1]{{\bf #1}}

\def\figref#1{figure~\ref{#1}}
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
\def\secref#1{section~\ref{#1}}
\def\Secref#1{Section~\ref{#1}}
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
\def\eqref#1{equation~\ref{#1}}
\def\Eqref#1{Equation~\ref{#1}}
\def\plaineqref#1{\ref{#1}}
\def\chapref#1{chapter~\ref{#1}}
\def\Chapref#1{Chapter~\ref{#1}}
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
\def\algref#1{algorithm~\ref{#1}}
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
\def\partref#1{part~\ref{#1}}
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}

\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\bm{1}{\bm{1}}
\newcommand{\mathcal{D}}{\mathcal{D}}
\newcommand{\mathcal{D_{\mathrm{valid}}}}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\mathcal{D_{\mathrm{test}}}}{\mathcal{D_{\mathrm{test}}}}

\def{\epsilon}{{\epsilon}}

\def{\textnormal{$\eta$}}{{\textnormal{$\eta$}}}
\def{\textnormal{a}}{{\textnormal{a}}}
\def{\textnormal{b}}{{\textnormal{b}}}
\def{\textnormal{c}}{{\textnormal{c}}}
\def{\textnormal{d}}{{\textnormal{d}}}
\def{\textnormal{e}}{{\textnormal{e}}}
\def{\textnormal{f}}{{\textnormal{f}}}
\def{\textnormal{g}}{{\textnormal{g}}}
\def{\textnormal{h}}{{\textnormal{h}}}
\def{\textnormal{i}}{{\textnormal{i}}}
\def{\textnormal{j}}{{\textnormal{j}}}
\def{\textnormal{k}}{{\textnormal{k}}}
\def{\textnormal{l}}{{\textnormal{l}}}
\def{\textnormal{n}}{{\textnormal{n}}}
\def{\textnormal{o}}{{\textnormal{o}}}
\def{\textnormal{p}}{{\textnormal{p}}}
\def{\textnormal{q}}{{\textnormal{q}}}
\def{\textnormal{r}}{{\textnormal{r}}}
\def{\textnormal{s}}{{\textnormal{s}}}
\def{\textnormal{t}}{{\textnormal{t}}}
\def{\textnormal{u}}{{\textnormal{u}}}
\def{\textnormal{v}}{{\textnormal{v}}}
\def{\textnormal{w}}{{\textnormal{w}}}
\def{\textnormal{x}}{{\textnormal{x}}}
\def{\textnormal{y}}{{\textnormal{y}}}
\def{\textnormal{z}}{{\textnormal{z}}}

\def{\mathbf{\epsilon}}{{\mathbf{\epsilon}}}
\def{\mathbf{\theta}}{{\mathbf{\theta}}}
\def{\mathbf{a}}{{\mathbf{a}}}
\def{\mathbf{b}}{{\mathbf{b}}}
\def{\mathbf{c}}{{\mathbf{c}}}
\def{\mathbf{d}}{{\mathbf{d}}}
\def{\mathbf{e}}{{\mathbf{e}}}
\def{\mathbf{f}}{{\mathbf{f}}}
\def{\mathbf{g}}{{\mathbf{g}}}
\def{\mathbf{h}}{{\mathbf{h}}}
\def{\mathbf{u}}{{\mathbf{i}}}
\def{\mathbf{j}}{{\mathbf{j}}}
\def{\mathbf{k}}{{\mathbf{k}}}
\def{\mathbf{l}}{{\mathbf{l}}}
\def{\mathbf{m}}{{\mathbf{m}}}
\def{\mathbf{n}}{{\mathbf{n}}}
\def{\mathbf{o}}{{\mathbf{o}}}
\def{\mathbf{p}}{{\mathbf{p}}}
\def{\mathbf{q}}{{\mathbf{q}}}
\def{\mathbf{r}}{{\mathbf{r}}}
\def{\mathbf{s}}{{\mathbf{s}}}
\def{\mathbf{t}}{{\mathbf{t}}}
\def{\mathbf{u}}{{\mathbf{u}}}
\def{\mathbf{v}}{{\mathbf{v}}}
\def{\mathbf{w}}{{\mathbf{w}}}
\def{\mathbf{x}}{{\mathbf{x}}}
\def{\mathbf{y}}{{\mathbf{y}}}
\def{\mathbf{z}}{{\mathbf{z}}}

\def{\textnormal{a}}{{\textnormal{a}}}
\def{\textnormal{b}}{{\textnormal{b}}}
\def{\textnormal{c}}{{\textnormal{c}}}
\def{\textnormal{d}}{{\textnormal{d}}}
\def{\textnormal{e}}{{\textnormal{e}}}
\def{\textnormal{f}}{{\textnormal{f}}}
\def{\textnormal{g}}{{\textnormal{g}}}
\def{\textnormal{h}}{{\textnormal{h}}}
\def{\textnormal{i}}{{\textnormal{i}}}
\def{\textnormal{j}}{{\textnormal{j}}}
\def{\textnormal{k}}{{\textnormal{k}}}
\def{\textnormal{l}}{{\textnormal{l}}}
\def{\textnormal{m}}{{\textnormal{m}}}
\def{\textnormal{n}}{{\textnormal{n}}}
\def{\textnormal{o}}{{\textnormal{o}}}
\def{\textnormal{p}}{{\textnormal{p}}}
\def{\textnormal{q}}{{\textnormal{q}}}
\def{\textnormal{r}}{{\textnormal{r}}}
\def{\textnormal{s}}{{\textnormal{s}}}
\def{\textnormal{t}}{{\textnormal{t}}}
\def{\textnormal{u}}{{\textnormal{u}}}
\def{\textnormal{v}}{{\textnormal{v}}}
\def{\textnormal{w}}{{\textnormal{w}}}
\def{\textnormal{x}}{{\textnormal{x}}}
\def{\textnormal{y}}{{\textnormal{y}}}
\def{\textnormal{z}}{{\textnormal{z}}}

\def{\mathbf{A}}{{\mathbf{A}}}
\def{\mathbf{B}}{{\mathbf{B}}}
\def{\mathbf{C}}{{\mathbf{C}}}
\def{\mathbf{D}}{{\mathbf{D}}}
\def{\mathbf{E}}{{\mathbf{E}}}
\def{\mathbf{F}}{{\mathbf{F}}}
\def{\mathbf{G}}{{\mathbf{G}}}
\def{\mathbf{H}}{{\mathbf{H}}}
\def{\mathbf{I}}{{\mathbf{I}}}
\def{\mathbf{J}}{{\mathbf{J}}}
\def{\mathbf{K}}{{\mathbf{K}}}
\def{\mathbf{L}}{{\mathbf{L}}}
\def{\mathbf{M}}{{\mathbf{M}}}
\def{\mathbf{N}}{{\mathbf{N}}}
\def{\mathbf{O}}{{\mathbf{O}}}
\def{\mathbf{P}}{{\mathbf{P}}}
\def{\mathbf{Q}}{{\mathbf{Q}}}
\def{\mathbf{R}}{{\mathbf{R}}}
\def{\mathbf{S}}{{\mathbf{S}}}
\def{\mathbf{T}}{{\mathbf{T}}}
\def{\mathbf{U}}{{\mathbf{U}}}
\def{\mathbf{V}}{{\mathbf{V}}}
\def{\mathbf{W}}{{\mathbf{W}}}
\def{\mathbf{X}}{{\mathbf{X}}}
\def{\mathbf{Y}}{{\mathbf{Y}}}
\def{\mathbf{Z}}{{\mathbf{Z}}}

\def{\textnormal{A}}{{\textnormal{A}}}
\def{\textnormal{B}}{{\textnormal{B}}}
\def{\textnormal{C}}{{\textnormal{C}}}
\def{\textnormal{D}}{{\textnormal{D}}}
\def{\textnormal{E}}{{\textnormal{E}}}
\def{\textnormal{F}}{{\textnormal{F}}}
\def{\textnormal{G}}{{\textnormal{G}}}
\def{\textnormal{H}}{{\textnormal{H}}}
\def{\textnormal{I}}{{\textnormal{I}}}
\def{\textnormal{J}}{{\textnormal{J}}}
\def{\textnormal{K}}{{\textnormal{K}}}
\def{\textnormal{L}}{{\textnormal{L}}}
\def{\textnormal{M}}{{\textnormal{M}}}
\def{\textnormal{N}}{{\textnormal{N}}}
\def{\textnormal{O}}{{\textnormal{O}}}
\def{\textnormal{P}}{{\textnormal{P}}}
\def{\textnormal{Q}}{{\textnormal{Q}}}
\def{\textnormal{R}}{{\textnormal{R}}}
\def{\textnormal{S}}{{\textnormal{S}}}
\def{\textnormal{T}}{{\textnormal{T}}}
\def{\textnormal{U}}{{\textnormal{U}}}
\def{\textnormal{V}}{{\textnormal{V}}}
\def{\textnormal{W}}{{\textnormal{W}}}
\def{\textnormal{X}}{{\textnormal{X}}}
\def{\textnormal{Y}}{{\textnormal{Y}}}
\def{\textnormal{Z}}{{\textnormal{Z}}}

\def{\bm{0}}{{\bm{0}}}
\def{\bm{1}}{{\bm{1}}}
\def{\bm{\mu}}{{\bm{\mu}}}
\def{\bm{\theta}}{{\bm{\theta}}}
\def{\bm{a}}{{\bm{a}}}
\def{\bm{b}}{{\bm{b}}}
\def{\bm{c}}{{\bm{c}}}
\def{\bm{d}}{{\bm{d}}}
\def{\bm{e}}{{\bm{e}}}
\def{\bm{f}}{{\bm{f}}}
\def{\bm{g}}{{\bm{g}}}
\def{\bm{h}}{{\bm{h}}}
\def{\bm{i}}{{\bm{i}}}
\def{\bm{j}}{{\bm{j}}}
\def{\bm{k}}{{\bm{k}}}
\def{\bm{l}}{{\bm{l}}}
\def{\bm{m}}{{\bm{m}}}
\def{\bm{n}}{{\bm{n}}}
\def{\bm{o}}{{\bm{o}}}
\def{\bm{p}}{{\bm{p}}}
\def{\bm{q}}{{\bm{q}}}
\def{\bm{r}}{{\bm{r}}}
\def{\bm{s}}{{\bm{s}}}
\def{\bm{t}}{{\bm{t}}}
\def{\bm{u}}{{\bm{u}}}
\def{\bm{v}}{{\bm{v}}}
\def{\bm{w}}{{\bm{w}}}
\def{\bm{x}}{{\bm{x}}}
\def{\bm{y}}{{\bm{y}}}
\def{\bm{z}}{{\bm{z}}}

\def{\alpha}{{\alpha}}
\def{\beta}{{\beta}}
\def{\epsilon}{{\epsilon}}
\def{\lambda}{{\lambda}}
\def{\omega}{{\omega}}
\def{\mu}{{\mu}}
\def{\psi}{{\psi}}
\def{\sigma}{{\sigma}}
\def{\theta}{{\theta}}
\def{a}{{a}}
\def{b}{{b}}
\def{c}{{c}}
\def{d}{{d}}
\def{e}{{e}}
\def{f}{{f}}
\def{g}{{g}}
\def{h}{{h}}
\def{i}{{i}}
\def{j}{{j}}
\def{k}{{k}}
\def{l}{{l}}
\def{m}{{m}}
\def{n}{{n}}
\def{o}{{o}}
\def{p}{{p}}
\def{q}{{q}}
\def{r}{{r}}
\def{s}{{s}}
\def{t}{{t}}
\def{u}{{u}}
\def{v}{{v}}
\def{w}{{w}}
\def{x}{{x}}
\def{y}{{y}}
\def{z}{{z}}

\def{\bm{A}}{{\bm{A}}}
\def{\bm{B}}{{\bm{B}}}
\def{\bm{C}}{{\bm{C}}}
\def{\bm{D}}{{\bm{D}}}
\def{\bm{E}}{{\bm{E}}}
\def{\bm{F}}{{\bm{F}}}
\def{\bm{G}}{{\bm{G}}}
\def{\bm{H}}{{\bm{H}}}
\def{\bm{I}}{{\bm{I}}}
\def{\bm{J}}{{\bm{J}}}
\def{\bm{K}}{{\bm{K}}}
\def{\bm{L}}{{\bm{L}}}
\def{\bm{M}}{{\bm{M}}}
\def{\bm{N}}{{\bm{N}}}
\def{\bm{O}}{{\bm{O}}}
\def{\bm{P}}{{\bm{P}}}
\def{\bm{Q}}{{\bm{Q}}}
\def{\bm{R}}{{\bm{R}}}
\def{\bm{S}}{{\bm{S}}}
\def{\bm{T}}{{\bm{T}}}
\def{\bm{U}}{{\bm{U}}}
\def{\bm{V}}{{\bm{V}}}
\def{\bm{W}}{{\bm{W}}}
\def{\bm{X}}{{\bm{X}}}
\def{\bm{Y}}{{\bm{Y}}}
\def{\bm{Z}}{{\bm{Z}}}
\def{\bm{\beta}}{{\bm{\beta}}}
\def{\bm{\Phi}}{{\bm{\Phi}}}
\def{\bm{\Lambda}}{{\bm{\Lambda}}}
\def{\bm{\Sigma}}{{\bm{\Sigma}}}

\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def{\tens{A}}{{\tens{A}}}
\def{\tens{B}}{{\tens{B}}}
\def{\tens{C}}{{\tens{C}}}
\def{\tens{D}}{{\tens{D}}}
\def{\tens{E}}{{\tens{E}}}
\def{\tens{F}}{{\tens{F}}}
\def{\tens{G}}{{\tens{G}}}
\def{\tens{H}}{{\tens{H}}}
\def{\tens{I}}{{\tens{I}}}
\def{\tens{J}}{{\tens{J}}}
\def{\tens{K}}{{\tens{K}}}
\def{\tens{L}}{{\tens{L}}}
\def{\tens{M}}{{\tens{M}}}
\def{\tens{N}}{{\tens{N}}}
\def{\tens{O}}{{\tens{O}}}
\def{\tens{P}}{{\tens{P}}}
\def{\tens{Q}}{{\tens{Q}}}
\def{\tens{R}}{{\tens{R}}}
\def{\tens{S}}{{\tens{S}}}
\def{\tens{T}}{{\tens{T}}}
\def{\tens{U}}{{\tens{U}}}
\def{\tens{V}}{{\tens{V}}}
\def{\tens{W}}{{\tens{W}}}
\def{\tens{X}}{{\tens{X}}}
\def{\tens{Y}}{{\tens{Y}}}
\def{\tens{Z}}{{\tens{Z}}}

\def{\mathcal{A}}{{\mathcal{A}}}
\def{\mathcal{B}}{{\mathcal{B}}}
\def{\mathcal{C}}{{\mathcal{C}}}
\def{\mathcal{D}}{{\mathcal{D}}}
\def{\mathcal{E}}{{\mathcal{E}}}
\def{\mathcal{F}}{{\mathcal{F}}}
\def{\mathcal{G}}{{\mathcal{G}}}
\def{\mathcal{H}}{{\mathcal{H}}}
\def{\mathcal{I}}{{\mathcal{I}}}
\def{\mathcal{J}}{{\mathcal{J}}}
\def{\mathcal{K}}{{\mathcal{K}}}
\def{\mathcal{L}}{{\mathcal{L}}}
\def{\mathcal{M}}{{\mathcal{M}}}
\def{\mathcal{N}}{{\mathcal{N}}}
\def{\mathcal{O}}{{\mathcal{O}}}
\def{\mathcal{P}}{{\mathcal{P}}}
\def{\mathcal{Q}}{{\mathcal{Q}}}
\def{\mathcal{R}}{{\mathcal{R}}}
\def{\mathcal{S}}{{\mathcal{S}}}
\def{\mathcal{T}}{{\mathcal{T}}}
\def{\mathcal{U}}{{\mathcal{U}}}
\def{\mathcal{V}}{{\mathcal{V}}}
\def{\mathcal{W}}{{\mathcal{W}}}
\def{\mathcal{X}}{{\mathcal{X}}}
\def{\mathcal{Y}}{{\mathcal{Y}}}
\def{\mathcal{Z}}{{\mathcal{Z}}}

\def{\mathbb{A}}{{\mathbb{A}}}
\def{\mathbb{B}}{{\mathbb{B}}}
\def{\mathbb{C}}{{\mathbb{C}}}
\def{\mathbb{D}}{{\mathbb{D}}}
\def{\mathbb{E}}{{\mathbb{E}}}
\def{\mathbb{F}}{{\mathbb{F}}}
\def{\mathbb{G}}{{\mathbb{G}}}
\def{\mathbb{H}}{{\mathbb{H}}}
\def{\mathbb{I}}{{\mathbb{I}}}
\def{\mathbb{J}}{{\mathbb{J}}}
\def{\mathbb{K}}{{\mathbb{K}}}
\def{\mathbb{L}}{{\mathbb{L}}}
\def{\mathbb{M}}{{\mathbb{M}}}
\def{\mathbb{N}}{{\mathbb{N}}}
\def{\mathbb{O}}{{\mathbb{O}}}
\def{\mathbb{P}}{{\mathbb{P}}}
\def{\mathbb{Q}}{{\mathbb{Q}}}
\def{\mathbb{R}}{{\mathbb{R}}}
\def{\mathbb{S}}{{\mathbb{S}}}
\def{\mathbb{T}}{{\mathbb{T}}}
\def{\mathbb{U}}{{\mathbb{U}}}
\def{\mathbb{V}}{{\mathbb{V}}}
\def{\mathbb{W}}{{\mathbb{W}}}
\def{\mathbb{X}}{{\mathbb{X}}}
\def{\mathbb{Y}}{{\mathbb{Y}}}
\def{\mathbb{Z}}{{\mathbb{Z}}}

\def{\Lambda}{{\Lambda}}
\def{A}{{A}}
\def{B}{{B}}
\def{C}{{C}}
\def{D}{{D}}
\def{E}{{E}}
\def{F}{{F}}
\def{G}{{G}}
\def{H}{{H}}
\def{I}{{I}}
\def{J}{{J}}
\def{K}{{K}}
\def{L}{{L}}
\def{M}{{M}}
\def{N}{{N}}
\def{O}{{O}}
\def{P}{{P}}
\def{Q}{{Q}}
\def{R}{{R}}
\def{S}{{S}}
\def{T}{{T}}
\def{U}{{U}}
\def{V}{{V}}
\def{W}{{W}}
\def{X}{{X}}
\def{Y}{{Y}}
\def{Z}{{Z}}
\def{\Sigma}{{\Sigma}}

\newcommand{\etens}[1]{\mathsfit{#1}}
\def{\etens{\Lambda}}{{\etens{\Lambda}}}
\def{\etens{A}}{{\etens{A}}}
\def{\etens{B}}{{\etens{B}}}
\def{\etens{C}}{{\etens{C}}}
\def{\etens{D}}{{\etens{D}}}
\def{\etens{E}}{{\etens{E}}}
\def{\etens{F}}{{\etens{F}}}
\def{\etens{G}}{{\etens{G}}}
\def{\etens{H}}{{\etens{H}}}
\def{\etens{I}}{{\etens{I}}}
\def{\etens{J}}{{\etens{J}}}
\def{\etens{K}}{{\etens{K}}}
\def{\etens{L}}{{\etens{L}}}
\def{\etens{M}}{{\etens{M}}}
\def{\etens{N}}{{\etens{N}}}
\def{\etens{O}}{{\etens{O}}}
\def{\etens{P}}{{\etens{P}}}
\def{\etens{Q}}{{\etens{Q}}}
\def{\etens{R}}{{\etens{R}}}
\def{\etens{S}}{{\etens{S}}}
\def{\etens{T}}{{\etens{T}}}
\def{\etens{U}}{{\etens{U}}}
\def{\etens{V}}{{\etens{V}}}
\def{\etens{W}}{{\etens{W}}}
\def{\etens{X}}{{\etens{X}}}
\def{\etens{Y}}{{\etens{Y}}}
\def{\etens{Z}}{{\etens{Z}}}

\newcommand{p_{\rm{data}}}{p_{\rm{data}}}
\newcommand{\hat{p}_{\rm{data}}}{\hat{p}_{\rm{data}}}
\newcommand{\hat{P}_{\rm{data}}}{\hat{P}_{\rm{data}}}
\newcommand{p_{\rm{model}}}{p_{\rm{model}}}
\newcommand{P_{\rm{model}}}{P_{\rm{model}}}
\newcommand{\tilde{p}_{\rm{model}}}{\tilde{p}_{\rm{model}}}
\newcommand{p_{\rm{encoder}}}{p_{\rm{encoder}}}
\newcommand{p_{\rm{decoder}}}{p_{\rm{decoder}}}
\newcommand{p_{\rm{reconstruct}}}{p_{\rm{reconstruct}}}

\newcommand{\laplace}{\mathrm{Laplace}}

\newcommand{\mathbb{E}}{\mathbb{E}}
\newcommand{\mathcal{L}}{\mathcal{L}}
\newcommand{\mathbb{R}}{\mathbb{R}}
\newcommand{\tilde{p}}{\tilde{p}}
\newcommand{\alpha}{\alpha}
\newcommand{\lambda}{\lambda}
\newcommand{\mathrm{rectifier}}{\mathrm{rectifier}}
\newcommand{\mathrm{softmax}}{\mathrm{softmax}}
\newcommand{\sigma}{\sigma}
\newcommand{\zeta}{\zeta}
\newcommand{D_{\mathrm{KL}}}{D_{\mathrm{KL}}}
\newcommand{\mathrm{Var}}{\mathrm{Var}}
\newcommand{\mathrm{SE}}{\mathrm{SE}}
\newcommand{\mathrm{Cov}}{\mathrm{Cov}}
\newcommand{L^0}{L^0}
\newcommand{L^1}{L^1}
\newcommand{L^2}{L^2}
\newcommand{L^p}{L^p}
\newcommand{L^\infty}{L^\infty}

\newcommand{\parents}{Pa}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\DeclareMathOperator{\sign}{sgn}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak

\section{Platform}
\label{sec:platform}

At a high level, {\sc \small TrojanZoo}\xspace consists of three main components as illustrated in Figure\mref{fig:framework}: \mcounter{i} the attack library that implements representative attacks, \mcounter{ii} the defense library that integrates state-of-the-art defenses, and \mcounter{iii} the analysis engine that, equipped with attack performance metrics, defense utility metrics, and feature-rich utility tools, is able to conduct unified and holistic evaluations across various attacks/defenses,

\begin{table*}[!ht]{\footnotesize
\centering
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{r|c|c|c|c|c}
\multirow{2}{*}{\bf Neural Backdoor Attack} & {\bf Architecture} & {\bf Trigger} & {\bf Training} & {\bf Fine-tuning} & {\bf Defense}
\\
& {\bf Modifiability} & {\bf Optimizability} & {\bf Controllability} & {\bf Survivability} & {\bf Adaptivity}\\
\hline
\hline
BadNet ({\sc Bn}\xspace)\mcite{badnet} & $\Circle$ & $\Circle$ & $\Circle$ & $\Circle$ & $\Circle$ \\
Embarrassingly Simple Backdoor ({\sc Esb}\xspace)\mcite{embarassingly-simple-backdoor} & $\CIRCLE$ & $\Circle$ & $\Circle$ & $\Circle$ & $\Circle$ \\
TrojanNN ({\sc Tnn}\xspace)\mcite{trojannn} & $\Circle$ & $\LEFTcircle$ & $\Circle$ & $\Circle$ & $\Circle$ \\
Reflection Backdoor ({\sc Rb}\xspace)\mcite{reflection-backdoor} & $\Circle$ & $\LEFTcircle$ & $\Circle$ & $\Circle$ & $\Circle$ \\
Targeted Backdoor ({\sc Tb}\xspace)\mcite{targeted-backdoor} & $\Circle$ & $\LEFTcircle$ & $\Circle$ & $\Circle$ & $\Circle$ \\
Dynamic Backdoor ({\sc Db}\xspace)\mcite{adaptive-trigger} & $\Circle$ & $\CIRCLE$ & $\Circle$ & $\Circle$ & $\Circle$ \\
Clean-Label Backdoor ({\sc Clb}\xspace)\mcite{clean-label-backdoor} & $\Circle$ & $\Circle$ & $\CIRCLE$ & $\Circle$ & $\Circle$ \\
Hidden Trigger Backdoor ({\sc Htb}\xspace)\mcite{hidden-trigger} & $\Circle$ & $\Circle$ & $\CIRCLE$ & $\Circle$ & $\Circle$ \\
Blind Backdoor ({\sc Bb}\xspace)\mcite{blind-backdoor}& $\Circle$ & $\Circle$ & $\CIRCLE$ & $\Circle$ & $\Circle$ \\
Latent Backdoor ({\sc Lb}\xspace)\mcite{latent-backdoor} & $\Circle$ & $\Circle$ & $\Circle$ & $\CIRCLE$ & $\Circle$ \\
Adversarial Backdoor Embedding ({\sc Abe}\xspace)\mcite{adv-backdoor} & $\Circle$ & $\Circle$ & $\Circle$ & $\Circle$ & $\CIRCLE$ \\
Input-Model Co-optimization ({\sc Imc}\xspace)\mcite{imc} & $\Circle$ & $\CIRCLE$ & $\Circle$ & $\CIRCLE$ & $\CIRCLE$ \\
\end{tabular}
\caption{\footnotesize Summary of representative neural backdoor attacks currently implemented in {\sc \small TrojanZoo}\xspace ($\CIRCLE$ -- full optimization, $\LEFTcircle$ -- partial optimization, $\Circle$ -- no optimization)}
\label{tab:attack_summary}
}
\end{table*}

In its current implementation, {\sc \small TrojanZoo}\xspace incorporates 12 attacks, 15 defenses, 6 attack performance metrics, and 10 defense utility metrics, which we systemize as follows.

\subsection{Attacks}

While neural backdoor attacks can be characterized by a range of aspects, here we focus on five key design choices by the adversary that directly impact attack performance.

\begin{mitemize}
\item {\em \underline{Architecture modifiability}} -- whether the attack is able to change the DNN architecture. Being allowed to modify both the architecture and the parameters enables a larger attack spectrum, but also renders the trojan model more susceptible to certain defenses (\textit{e.g.}\xspace,\,model specification checking).

\item {\em \underline{Trigger optimizability}} -- whether the attack uses a fixed, pre-defined trigger or optimizes it during crafting the trojan model. Trigger optimization often leads to stronger attacks with respect to given desiderata (\textit{e.g.}\xspace,\,trigger stealthiness).

\item {\em \underline{Training controllability}} -- whether the adversary has control over the training of trojan models. Under the setting that the victim user controls the model training, the adversary may influence the training only through injecting poisoning data or compromising the training code.

\item {\em \underline{Fine-tuning survivability}} -- whether the backdoor remains effective if the model is fine-tuned. A pre-trained model is often composed with a classifier and fine-tuned using the data from the downstream task. It is desirable to ensure that the backdoor remains effective after fine-tuning.

\item {\em \underline{Defense adaptivity}} -- whether the attack is optimizable to evade possible defenses. For the attack to be effective, it is essential to optimize the evasiveness of the trojan model and the trigger input with respect to the deployed defenses.
\end{mitemize}

Table\mref{tab:attack_summary} summarizes the representative neural backdoor attacks currently implemented in {\sc \small TrojanZoo}\xspace, which are characterized along the above five dimensions.

\vspace{1pt}
{\bf Non-optimization --} {{\sc Bn}\xspace}\mcite{badnet}, as the simplest attack, pre-defines a trigger $r$ (\textit{i.e.}\xspace,\,shape, position, and pattern), generates trigger inputs $\{ (x\oplus r, t)\}$, and forges the trojan model $\ssup{f}{*}$ by re-training a benign model $f$ with such data.

\vspace{1pt}
{\bf Architecture modifiability --} {{\sc Esb}\xspace}\mcite{embarassingly-simple-backdoor} modifies $f$'s architecture by adding a module which overwrites the prediction as $t$ if $r$ is recognized. Without disturbing $f$'s original configuration, $\ssup{f}{*}$ retains $f$'s predictive power on clean inputs.

\vspace{1pt}
{\bf Trigger optimizability --} {{\sc Tnn}\xspace}\mcite{trojannn} fixes $r$'s shape and position, optimizes its pattern to activate neurons rarely activated by clean inputs in pre-processing, and then forges $\ssup{f}{*}$ by re-training $f$ in a manner similar to {\sc Bn}\xspace.

{{\sc Rb}\xspace}\mcite{reflection-backdoor} optimizes trigger stealthiness by defining $r$ as the physical reflection of a clean image $\ssup{x}{r}$ (selected from a pool): $r = \ssup{x}{r} \otimes k$, where $k$ is a convolution kernel, and $\otimes$ is the convolution operator.

{{\sc Tb}\xspace}\mcite{targeted-backdoor} randomly generates $r$'s position in training, which makes $\ssup{f}{*}$ effective regardless of $r$'s position and allows the adversary to optimize $r$'s stealthiness by placing it at the most plausible position (\textit{e.g.}\xspace,\,an eyewear watermark over eyes).

{{\sc Db}\xspace}\mcite{adaptive-trigger} uses a generative network to generate $r$'s pattern and position dynamically, which is trained jointly with $\ssup{f}{*}$.

\vspace{1pt}
{\bf Training controllability --} {{\sc Clb}\xspace}\mcite{clean-label-backdoor} assumes the setting that the adversary forges $\ssup{f}{*}$ via polluting the training data. To evade possible filtering, {\sc Clb}\xspace generates (via either adversarial perturbation or generative networks) stealthy poisoning inputs that appear to be consistent with their labels.

{{\sc Htb}\xspace}\mcite{hidden-trigger} generates stealthy poisoning inputs that are close to trigger inputs in the feature space, but are correctly labeled (to human inspection) and do not contain visible triggers.

{{\sc Bb}\xspace}\mcite{blind-backdoor} assumes the setting that the adversary forges $\ssup{f}{*}$ via compromising the code of computing the loss function.

\vspace{1pt}
{\bf Fine-tuning survivability --} {{\sc Lb}\xspace}\mcite{latent-backdoor} accounts for the impact of downstream fine-tuning by optimizing $f$ with respect to latent representations rather than final predictions. Specifically, it instantiates \meq{eq:opt} with the following loss function:
$\ell(f(x \oplus r), t) = \Delta(f(x \oplus r), \phi_t)$,
where $\Delta$ measures the difference of two latent representations and $\phi_t$ denotes the representation of class $t$, defined as $\phi_t = \arg\min_{\phi} {\mathbb{E}}_{(x, t) \in \mathcal{T}} [ \Delta ( f(x), \phi_t) ] $.

\vspace{1pt}
{\bf Defense adaptivity --} {{\sc Abe}\xspace}\mcite{adv-backdoor} accounts for possible defenses in forging $\ssup{f}{*}$. In solving \meq{eq:opt}, {\sc Abe}\xspace also optimizes the indistinguishability of the latent representations of trigger and clean inputs. Specifically, it uses a discriminative network $d$ to predict the representation of a given input $x$ as trigger or clean. Formally, the loss is defined as $\Delta(d\circ f(x), b(x))$, where $b(x)$ encodes whether $x$ is trigger or clean, while $\ssup{f}{*}$ and $d$ are trained using an adversarial learning framework\mcite{Goodfellow:2014:nips}.

\vspace{1pt}
{\bf Co-optimization --} {{\sc Imc}\xspace}\mcite{imc} is motivated by the mutual-reinforcement effect between $r$ and $\ssup{f}{*}$: optimizing one may greatly amplify the effectiveness of the other. Instead of solving \meq{eq:opt} by first pre-defining $r$ and then optimizing $\ssup{f}{*}$, {\sc Imc}\xspace optimizes $r$ and $\ssup{f}{*}$ jointly, which enlarges the search spaces for $r$ and $\ssup{f}{*}$, leading to attacks satisfying multiple desiderata (\textit{e.g.}\xspace,\,fine-tuning survivability and defense adaptivity).

\begin{table*}[!ht]{\footnotesize
    \centering
    \renewcommand{\arraystretch}{1.2}
    \setlength{\tabcolsep}{2pt}
    \begin{tabular}{r|c|c|c|c|c|c|l}
    \multirow{2}{*}{\bf Neural Backdoor Defense} & \multirow{2}{*}{\bf Category} & \multicolumn{2}{c|}{\bf Mitigation} & \multicolumn{3}{c|}{\bf Detection Target} & \multirow{2}{*}{\bf Design Rationale} \\
    \cline{3-7}
    & & {\bf Input} & {\bf Model} & {\bf Input} & {\bf Model} & {\bf Trigger} & \\
    \hline
    \hline
    Randomized-Smoothing ({\sc Rs}\xspace)\mcite{randomized-smoothing} & \multirow{3}{*}{\minitab[c]{Input\\Reformation}} & $\checkmark$ & & & & & ${\mathcal{A}}$'s fidelity ($x$'s and $\ssup{x}{*}$'s surrounding class boundaries) \\
    Down-Upsampling ({\sc Du}\xspace)\mcite{feature-squeeze} & & $\checkmark$ & & & & & ${\mathcal{A}}$'s fidelity ($x$'s and $\ssup{x}{*}$'s high-level features) \\
    Manifold-Projection ({\sc Mp}\xspace)\mcite{magnet} & & $\checkmark$ & & & & & ${\mathcal{A}}$'s fidelity ($x$'s and $\ssup{x}{*}$'s manifold projections) \\
    \hline
    Activation-Clustering ({\sc Ac}\xspace)\mcite{active-clustering} & \multirow{4}{*}{\minitab[c]{Input\\Filtering}} & & & $\checkmark$ & & & distinct activation patterns of $\{x\}$ and $\{\ssup{x}{*}\}$ \\
    Spectral-Signature ({\sc Ss}\xspace)\mcite{spectral-signature} & & & & $\checkmark$ & & & distinct activation patterns of $\{x\}$ and $\{\ssup{x}{*}\}$ (spectral space) \\
    STRIP ({\sc Strip}\xspace)\mcite{strip} & & & & $\checkmark$ & & & distinct self-entropy of $x$'s and $\ssup{x}{*}$'s mixtures with clean inputs \\
    NEO ({\sc Neo}\xspace)\mcite{neo} & & & & $\checkmark$ & & & sensitivity of $\ssup{f}{*}$'s prediction to trigger perturbation \\
    Februus ({\sc Februus}\xspace)\mcite{februus} & & & & $\checkmark$ & & $\checkmark$ & sensitivity of $\ssup{f}{*}$'s prediction to trigger perturbation \\
    \hline
    Adversarial-Retraining ({\sc Ar}\xspace)\mcite{madry:iclr:2018} & Model & & $\checkmark$ & & & & ${\mathcal{A}}$'s fidelity ($x$'s and $\ssup{x}{*}$'s surrounding class boundaries) \\
    Fine-Pruning ({\sc Fp}\xspace)\mcite{fine-pruning} & Sanitization & & $\checkmark$ & & & & ${\mathcal{A}}$'s use of neurons rarely activated by clean inputs \\
    \hline
    NeuralCleanse ({\sc Nc}\xspace)\mcite{neural-cleanse} & \multirow{6}{*}{\minitab[c]{Model\\Inpsection}} & & & & $\checkmark$ & $\checkmark$ & abnormally small perturbation from other classes to $t$ in $f$\\
    DeepInspect ({\sc Di}\xspace)\mcite{deep-inspect} & & & & & $\checkmark$ & $\checkmark$ & abnormally small perturbation from other classes to $t$ in $\ssup{f}{*}$\\
    TABOR ({\sc Tabor}\xspace)\mcite{tabor} & & & & & $\checkmark$ & $\checkmark$ & abnormally small perturbation from other classes to $t$ in $f$ \\
    NeuronInspect ({\sc Ni}\xspace)\mcite{neuron-inspect} & & & & & $\checkmark$ & & distinct explanations of $f$ and $\ssup{f}{*}$ with respect to clean inputs \\
    ABS ({\sc Abs}\xspace)\mcite{abs} & & & & &$\checkmark$ &$\checkmark$ & ${\mathcal{A}}$'s use of neurons elevating $t$'s prediction \\

    \end{tabular}
    \caption{\footnotesize Summary of representative neural backdoor defenses currently implemented in {\sc \small TrojanZoo}\xspace (${\mathcal{A}}$ -- backdoor attack, $x$ -- clean input, $x^*$ -- trigger input, $f$ -- benign model, $f^*$ -- trojan model, $t$ -- target class) \label{tab:defense_summary}}}
    \end{table*}

\subsection{Attack Performance Metrics}
\label{sec:attack-metric}

Currently, {\sc \small TrojanZoo}\xspace incorporates 6 metrics to assess the effectiveness, evasiveness, and transferability of given attacks.

\vspace{1pt}
{\em \underline{Attack success rate}} ({\em \small ASR}\xspace) -- which measures the likelihood that trigger inputs are classified to the target class $t$:
\begin{align}
\textrm{{\em \small ASR}\xspace} = \frac{\textrm{\small \# successful trials}}{\textrm{\small \# total trials}}
\end{align}
Typically, higher {\em \small ASR}\xspace indicates more effective attacks.

\vspace{1pt}
{\em \underline{Trojan misclassification confidence}} ({\em \small TMC}\xspace) -- which is the average confidence score assigned to class $t$ of trigger inputs in successful attacks. Intuitively, {\em \small TMC}\xspace complements {\em \small ASR}\xspace and measures attack efficacy from another perspective.

\vspace{1pt}
{\em \underline{Clean accuracy drop}} ({\em \small CAD}\xspace) -- which measures the difference of the classification accuracy of two systems built upon the benign model and its trojan counterpart, respectively; {\em \small CAD}\xspace measures the attack specificity (\textit{cf}.\xspace\,Figure\mref{fig:triangle}), that is, whether the attack directs its influence to trigger inputs only.

\vspace{1pt}
{\em \underline{Clean classification confidence}} ({\em \small CCC}\xspace) -- which is the average confidence assigned to the ground-truth classes of clean inputs; {\em \small CCC}\xspace complements {\em \small CAD}\xspace by measuring attack specificity from the perspective of classification confidence.

\vspace{1pt}
{\em \underline{Efficacy-specificity AUC}} ({\em \small AUC}\xspace) -- which quantifies the aggregated trade-off between attack efficacy (measured by {\em \small ASR}\xspace) and attack specificity (measured by {\em \small CAD}\xspace). As revealed in\mcite{imc}, there exists an intricate balance: at a proper cost of specificity, it is possible to significantly improve efficacy, and vice versa; {\em \small AUC}\xspace measures the area under the {\em \small ASR}\xspace-{\em \small CAD}\xspace curve. Intuitively, smaller {\em \small AUC}\xspace implies a more significant trade-off effect.

\vspace{1pt}
{\em \underline{Neuron-separation ratio}} ({\em \small NSR}\xspace) -- which measures the intersection between neurons activated by clean and trigger inputs. In the penultimate layer of the model, we find ${\mathcal{N}}_\mathrm{c}$ and ${\mathcal{N}}_\mathrm{t}$, the top-$k$ active neurons with respect to clean and trigger inputs, respectively, and calculate their jaccard index:
\begin{align}
\textrm{{\em \small NSR}\xspace} = 1- |{\mathcal{N}}_\mathrm{t} \cap {\mathcal{N}}_\mathrm{c}|/|{\mathcal{N}}_\mathrm{t} \cup {\mathcal{N}}_\mathrm{c}|
\end{align}
Intuitively, {\em \small NSR}\xspace compares the neural activation patterns of clean and trigger inputs.

\subsection{Defenses}

The existing defenses against neural backdoor attacks, according to their strategies, can be categorized as:
\begin{mitemize}
\item {\em \underline{Input reformation}} -- which, before feeding an incoming input to the system, first reforms it to mitigate the influence of the potential trigger, yet without explicitly detecting whether it is a trigger input.
It typically exploits the high fidelity of an attack ${\mathcal{A}}$, that is, ${\mathcal{A}}$ tends to retain the perceptual similarity of a clean input $x$ and its trigger counterpart $\ssup{x}{*}$.

\item {\em \underline{Input filtering}} -- which detects whether an incoming input is embedded with a trigger and possibly recovers the clean input. It typically distinguishes clean and trigger inputs using their distinct characteristics.

\item {\em \underline{Model sanitization}} -- which, before using a pre-trained model $f$, sanitizes it to mitigate the potential backdoor, yet without explicitly detecting whether $f$ is trojaned.

\item {\em \underline{Model inspection}} -- which determines whether $f$ is a trojan model and, if so, recovers the target class and the potential trigger, at the model checking stage.

\end{mitemize}

Note that here we focus on the setting of transfer learning or outsourced training, which precludes certain other defenses such as purging poisoning training data\mcite{certified-purging}. Table\mref{tab:defense_summary} summarizes the 15 representative defenses currently implemented in {\sc \small TrojanZoo}\xspace, which are detailed below.

\vspace{1pt}
{\bf Input reformation --} {{\sc Rs}\xspace}\mcite{randomized-smoothing} exploits the premise that ${\mathcal{A}}$ retains the similarity of $x$ and $\ssup{x}{*}$ in terms of their surrounding class boundaries and classifies an input by averaging the predictions within its vicinity (via adding Gaussian noise).

{{\sc Du}\xspace}\mcite{feature-squeeze} exploits the premise that ${\mathcal{A}}$ retains the similarity of $x$ and $\ssup{x}{*}$ in terms of their high-level features while the trigger $r$ is typically not perturbation-tolerant. By downsampling and then upsampling $\ssup{x}{*}$, it is possible to mitigate $r$'s influence.

{{\sc Mp}\xspace}\mcite{magnet} exploits the premise that ${\mathcal{A}}$ retains the similarity of $x$ and $\ssup{x}{*}$ in terms of their projections to the data manifold. To this end, it trains an autoencoder to learn an approximate manifold, which projects $\ssup{x}{*}$ to the manifold.

\vspace{1pt}
{\bf Input filtering --} {{\sc Ac}\xspace}\mcite{active-clustering} distinguishes clean and trigger inputs by clustering their latent representations. While {\sc Ac}\xspace is also applicable for purging poisoning data, we consider its use as an input filtering method at inference time. {{\sc Ss}\xspace}\mcite{spectral-signature} exploits the similar property in the spectral space.

{{\sc Strip}\xspace}\mcite{strip} mixes a given input with a clean input and measures the self-entropy of its prediction. If the input is trigger-embedded, the mixture remains dominated by the trigger and tends to be misclassified, resulting in low self-entropy.

{{\sc Neo}\xspace}\mcite{neo} detects a trigger input by searching for a position, if replaced by a ``blocker'', changes its prediction, and uses this substitution to recover its original prediction. {{\sc Februus}\xspace}\mcite{februus} exploits the same property but uses a generative network to generate the substitution blocker.

\vspace{1pt}
{\bf Model sanitization --} By treating trigger inputs as one type of adversarial inputs, {{\sc Ar}\xspace}\mcite{madry:iclr:2018} applies adversarial training over the pre-trained model to improves its robustness to backdoor attacks. {{\sc Fp}\xspace}\mcite{fine-pruning} uses the property that the attack exploits spare model capacity. It thus prunes rarely used neurons and then applies fine-tuning to defend against pruning-aware attacks.

\vspace{1pt}
{\bf Model inspection --} Given a model $f$, {{\sc Nc}\xspace}\mcite{neural-cleanse} searches for potential triggers in each class $t$. If $t$ is trigger-embedded, the minimum perturbation required to change the predictions of the inputs in other classes to $t$ is abnormally small. {{\sc Di}\xspace}\mcite{deep-inspect} follows a similar pipeline but uses a generative network to generate trigger candidates. {{\sc Tabor}\xspace}\mcite{tabor} extends {\sc Nc}\xspace by adding a new regularizer to control the trigger search space.

{{\sc Ni}\xspace}\mcite{neuron-inspect} exploits the property that the explanation heatmaps of benign and trojan models manifest distinct characteristics. Using the features extracted from such heatmaps, {\sc Ni}\xspace detects trojan models as outliers.

{{\sc Abs}\xspace}\mcite{abs} first inspects $f$ to sift out abnormal neurons with large elevation difference (\textit{i.e.}\xspace,\,active only with respect to one specific class) and identifies triggers by maximizing abnormal neuron activation while preserving normal neuron behaviors.

\subsection{Defense Utility Metrics}
\label{sec:defense-metric}

Currently, {\sc \small TrojanZoo}\xspace incorporates 10 metrics to evaluate the robustness, utility-preservation, and genericity of given defenses. The metrics are tailored to the objectives of each defense category (\textit{e.g.}\xspace,\,trigger input detection).
For ease of exposition, below we consider the performance of a given defense ${\mathcal{D}}$ with respect to a given attack ${\mathcal{A}}$.

\vspace{1pt}
{\em \underline{Attack rate deduction}} ({\em \small ARD}\xspace) -- which measures the difference of ${\mathcal{A}}$'s {\em \small ASR}\xspace before and after ${\mathcal{D}}$. Intuitively, {\em \small ARD}\xspace indicates ${\mathcal{D}}$'s impact on ${\mathcal{A}}$'s efficacy. Intuitively, larger {\em \small ARD}\xspace indicates more effective defense. We also use ${\mathcal{A}}$'s {\em \small TMC}\xspace to measure ${\mathcal{D}}$'s influence on the classification confidence of trigger inputs.

\vspace{1pt}
{\em \underline{Clean accuracy drop}} ({\em \small CAD}\xspace) -- which measures the difference of the {\em \small ACC}\xspace of clean inputs before and after ${\mathcal{D}}$ is applied. It measures ${\mathcal{D}}$'s impact on the system's normal functionality. Note that {\em \small CAD}\xspace here is defined differently from its counterpart in attack performance metrics. We also use {\em \small CCC}\xspace to measure ${\mathcal{D}}$'s influence on the classification confidence of clean inputs.

\vspace{1pt}
{\em \underline{True positive rate}} ({\em \small TPR}\xspace) -- which, for input-filtering methods, measures the performance of detecting trigger inputs.
\begin{align}
\textrm{{\em \small TPR}\xspace} = \frac{\textrm{\small \# successfully detected trigger inputs}}{\textrm{\small \# total trigger inputs}}
\end{align}
Correspondingly, we use false positive rate ({\em \small FPR}\xspace) to measure the error of misclassifying clean inputs as trigger inputs.

\vspace{1pt}
{\em \underline{Anomaly index value}} ({\em \small AIV}\xspace) -- which measures the anomaly of trojan models in model-inspection defenses. Most existing methods (\textit{e.g.}\xspace,\,\mcite{neural-cleanse,deep-inspect,tabor,abs}) formalize finding trojan models as outlier detection: each class $t$ is associated with a score (\textit{e.g.}\xspace,\,minimum perturbation); if its score significantly deviates from others, $t$ is considered to contain a backdoor. {\em \small AIV}\xspace, the absolute deviations from median normalized by median absolute deviation ({\em \small MAD}\xspace), provide a reliable measure for such dispersion. Typically, $t$ with {\em \small AIV}\xspace larger than 2 has over 95\% probability of being anomaly.

\vspace{1pt}
{\em \underline{Mask L$_1$ norm}} ({\em \small MLN}\xspace) -- which measures the $\ell_1$-norm of the triggers recovered by model-inspection methods.

\vspace{1pt}
{\em \underline{Mask jaccard similarity}} ({\em \small MJS}\xspace) -- which further measures the intersection between the recovered trigger and the ground-truth trigger (injected by the adversary). Let $\ssup{m}{o}$ and $\ssup{m}{r}$ be the masks of original and recovered triggers. We define {\em \small MJS}\xspace as the Jaccard similarity of $\ssup{m}{o}$ and $\ssup{m}{r}$ :
\begin{align}
\textrm{\small {\em \small MJS}\xspace} = | O(\ssup{m}{o}) \cap O(\ssup{m}{r})|/|O(\ssup{m}{o}) \cup O(\ssup{m}{r})|
\end{align}
where $O(m)$ denotes the set of non-zero elements in $m$.

\vspace{1pt}
{\em \underline{Average running time}} ({\em \small ART}\xspace) -- which measures the overhead of ${\mathcal{D}}$. For model sanitization or inspection, which is performed offline, {\em \small ART}\xspace is measured as the running time per model; while for input filtering or reformation, which is executed online, {\em \small ART}\xspace is measured as the execution time per input.