Current artificial neural networks (ANNs) can perform and excel at a variety of tasks ranging from image classification to spam detection through training on large datasets of labeled data. While the trained network usually performs well on similar testing data, certain inputs that differ even slightly from the training data may trigger unpredictable behavior. Due to this limitation, it is possible to generate inputs with very small designed perturbations that can result in misclassification. These adversarial attacks present a security risk to deployed ANNs and indicate a divergence between how ANNs and humans perform classification. Humans are robust at behaving in the presence of noise and are capable of correctly classifying objects that are occluded, blurred, or otherwise distorted. It has been hypothesized that sleep promotes generalization and improves robustness against noise in animals and humans. In this work, we utilize a biologically inspired sleep phase in ANNs and demonstrate the benefit of sleep on defending against adversarial attacks as well as increasing ANN classification robustness. We compare the sleep algorithm's performance on various robustness tasks with two previously proposed adversarial defenses, defensive distillation and fine-tuning. We report an increase in robustness after sleep to adversarial attacks as well as to general image distortions for three datasets: MNIST, CUB200, and a toy dataset. Overall, these results demonstrate the potential for biologically inspired solutions to solve existing problems in ANNs and guide the development of more robust, human-like ANNs.
Disclosure on reviewer's experience: I am not an expert on adversarial attack methods or defenses, but I am well read in the general literature on robustness and uncertainty in deep neural networks.
The authors present a biologically inspired sleep algorithm for artificial neural networks (ANNs) that aims to improve their generalization and robustness in the face of noisy or malicious inputs. They hypothesize that "sleep" could aid in generalization by decorrelating noisy hidden states and reducing the overall impact of imperceptible perturbations of the input space. The proposed sleep algorithm broadly involves 1) converting the trained ANN to a "spike" neural network (SNN), 2) converting the input signal (pixels) to a Poisson distributed "spike train" where brighter pixels have higher firing rates than darker pixels, 3) propagating the neuronal spikes through the SNN, updating weights based on a simplified version of spike-timing-dependent plasticity (STDP), and 4) converting the network back to an ANN after the sleep phase has finished. They present a detailed comparative study spanning three datasets, four types of adversarial attacks and distortions, and two other baseline defense mechanisms, in which they demonstrate significant improvements (in some cases) of the sleep algorithm over the baselines.
The core concept behind the authors' work is novel and interesting, and the experimental design is thorough and well controlled. Although the results are (I would argue) somewhat mixed, they are nonetheless positive enough to encourage more work in applying "sleep" and other relevant ideas from neuroscience to the problem of robustness in deep neural networks. I have some questions and concerns which I will detail per-section below, but overall, I believe that this paper is a valuable contribution to the literature and should be accepted once the authors have made a few necessary revisions.
Section 1: Introduction
"We report positive results for four types of adversarial attacks tested on three different datasets (MNIST, CUB200, and a toy dataset) ..."
It's debatable whether or not the results from the CUB-200 dataset are positive. The sleep algorithm fails to outperform the baselines for each attack type (except for an almost negligible advantage in accuracy on JSMA) and barely even outperforms the control network in most cases (2/4 attacks it actually underperforms the control). I think the authors should consider rephrasing this statement to better reflect the actual results.
Section 2: Adversarial Attacks and Distortions
FGSM: The notation used here is somewhat inconsistent with the source paper. Goodfellow et al use epsilon to denote what I think the authors call eta, and call the second term, epsilon*sign(grad(J)), eta. Furthermore, the authors state that "this represents the direction to change each pixel in the original input in order to decrease the loss function." But this doesn't make sense. An adversary should want to *increase* the loss function enough to cause a misclassification. Goodfellow et al use this expression to formulate a L1-like regularization term and describe the training procedure "minimizing the worst case error when the data is perturbed by an adversary", which seems more sensible. This section should be rewritten to be more consistent with the source.
Section 3: Adversarial defenses
Regarding distillation: "We use T=50 to compare with the sleep algorithm"
The authors should elaborate a bit more on the reasoning for this choice. It seems very arbitrary.
Sectioin 4: Sleep algorithm
1. Algorithm 1: Why is line 9 inside of the for loop? It doesn't seem to be at all dependent on t. One would expect the input to only need to be converted once. Additionally, in lines 11-13, the l's in W(l,l-1) and similar should be unbolded. It's confusing that the format changes (unless I am missing something and it's actually a different variable).
2. Spike trains should be more rigorously defined, preferably with formalized notation. It's a bit unclear exactly what they are from the current text. Are they just parameters for a Poisson? Or outputs from a poison over T time steps? Or something else?
3. "weights are scaled by a parameter to induce high firing rates in later layers"
It would be good to include more details on this parameter, how the values are chosen, and the intuition behind this idea. I assume it's because of higher level feature representations in later layers of deep neural networks.
Section 5: Results
1. It's confusing that sometimes accuracy refers to classification accuracy and sometimes adversarial attack accuracy. I would recommend assigning a different name to the latter, or making sure that a qualifier precedes every reference to "accuracy" in this section.
2. In the second section of the results table (which is missing a label), why is the JSMA value for Defensive Distillation bolded? The distance measures for both the control network and for fine-tuning are higher. It seems like fine-tuning should be the one bolded.
3. Figure 1: caption is incorrect; it states "adversarial attack accuracy" and it should be "classification accuracy", otherwise the plots make no sense.
4. "we observe that in the Patches and CUB-200 dataset, sleep has beneficial results in moving the accuracy function above the other defense methods"
It should be noted that this is only true for eta < 0.1. After that, sleep and the control both converge to 50% accuracy. Also this sentence should be reworded to be less visual and more quantitative (e.g. sleep tends to have higher median accuracy scores than the other methods for eta < 0.1).
5. "We observe that performance continued to drop after a sufficiently large amount of noise was added"
More than that, the other methods converged to a small band of accuracy values; sleep continued to deteriorate. This is a significant difference. It would be a good idea to re-run this experiment with a binary classification problem (e.g. only two digits of MNIST) and see if this phenomenon still occurs. Then, the noisy sleep classifier predictions could simply be inverted to get improved accuracy scores.
6. In the analysis of JSMA, as noted before,, it's rather dubious to claim that sleep had any kind of significant effect on the attack success rate (or distance) for CUB-200. I would rewrite this section to better represent the results.
7. Figure 2 formatting: Legend is overflowing out of the first figure. Additionally, the legend colors should be made to match across all three figures, and the legend should either appear in all three (if necessary for some reason) or only in one.
8. Figure 2: The caption is incomplete and possibly incorrect. It's not clear why the first and last figures differ from each other, and the caption does not indicate this. The caption also only mentions two datasets, even though it says "for the following three datasets".
Appendix:
General formatting needs improvement. A lot of figures are off-centered, text misaligned, missing axis labels, etc.