As deep neural networks have become the state of the art for solving complex reinforcement learning tasks, susceptibility to perceptual adversarial examples have become a concern. The transferability of adversarial examples is known to enable attacks capable of tricking the agent into bad states. In this work we demonstrate a simple poisoning attack able to keep deep RL from learning, and into fooling it when trained with defense methods commonly used for classification tasks. We then propose an algorithm called DadQN, based on deep Q-networks, which enables the use of stronger defenses, including defenses enabling the first ever on-line robustness certification of a deep RL agent.
The goal of this paper is to train deep RL agents that perform well both in the presence and absence of adversarial attacks at training and test time. To achieve this, this paper proposes using policy distillation. The approach, Distilled Agent DQN (DaDQN), consists of: (1) a "teacher" neural network trained in the same way as DQN, and (2) a "student" network trained with supervised learning to match the teacher’s outputs. Adversarial defenses are only applied to the student network, so as to not impact the learning of Q-values by the teacher network. At test time, the student network is deployed.
This idea of separating the learning of Q-values from the incorporation of adversarial defenses is promising. One adversarial defense considered in the paper is adversarial training -- applying small FGSM perturbations to inputs before they are given to the network. In a sense, the proposed approach is the correct way of doing adversarial training in deep RL. Unlike in supervised learning, there is no ground truth for the correct action to take. But by treating the teacher's output (for an unperturbed input) as ground truth, the student network can more easily learn the correct Q-values for the corresponding perturbed input.
The experimental results support the claim that applying adversarial training to DaDQN leads to agents that perform well at test time, both in the presence and absence of adversarial attacks. Without this teacher-student separation, incorporating adversarial training severely impairs learning (Table 2, DQN Def column). This separation also enables training the student network with provably robust training.
However, I have a few significant concerns regarding this paper. The first is regarding the white-box poisoning attack that this paper proposes, called Untargeted Q-Poisoning (UQP). This is not a true poisoning attack, since it attacks not just at training time, but also at test time. Also, the choice of adding the *negative* of the FGSM perturbation during training time is not clearly justified. Why not just use FGSM perturbations? The reason given in the paper is that this reinforces the choice of the best action w.r.t. the learned Q-values, to give the illusion of successful training -- but why is this illusion important, and is this illusion actually observed during training time? What are the scores obtained at the end of training? Table 1 only reports test-time scores.
In addition, although most of the paper is written clearly, the experiment section is confusing. I have the following major questions:
- What is the attack Atk (Section 4.3) -- is it exactly the same as the defense Def, except the perturbations are now stored in the replay buffer? Are attack and defense perturbations applied at every timestep?
- In Section 4.2, when UQP is applied, is it attacking both at training and at test time? Given the definition of UQP (Section 2.4), the answer would be yes. If that’s the case, then the "none" row in Table 1 is misleading, since there actually is a test time attack.
The experiments could also be more thorough. For instance, is the adversarial training defense still effective when the FGSM \epsilon used in test time attacks is smaller or larger? Also, how important is it that the student network chooses actions during training time, rather than the teacher network? An ablation study would be helpful here.
Overall, although the algorithmic novelty is promising, it is relatively minor. Due to this, and the weaknesses mentioned above, I don't think this paper is ready for publication.
Minor comments / questions:
- Tables 1 and 2 should report 95% confidence intervals or the standard error.
- It’s strange to apply the attack to the entire 4-stack of consecutive frames used (i.e., the observations from the last four timesteps); it would make more sense if the attack only affected the current frame.
- For adversarial training, what probability p (Section 3.2) is used in the experiments?
- In Section 4.2, what does “weighted by number of frames” mean?
- In which experiments (if any) is NoisyNet used? Section 4.1 mentions it is disabled, and \epsilon-greedy exploration is used instead. But I assume it’s used somewhere, because it’s described when explaining the DaDQN approach (Section 3.1).