Deep reinforcement learning (DRL) has achieved great success in various applications, such as playing computer games and controlling robotic manipulations. However, recent studies show that machine learning models are vulnerable to adversarial examples, which are carefully crafted instances that aim to mislead these models to make arbitrarily incorrect prediction. DRL models have been attacked by adding perturbations to observations. While such observation based attack is only one aspect of potential attacks on DRL, other forms of attacks require further analysis which are more practical such as manipulating environment dynamics. Therefore, we propose to understand the vulnerabilities of DRL from various perspectives and provide a thorough taxonomy of adversarial attacks against DRL. We also conduct the first set of experiments on the unexplored parts within the taxonomy. In addition to current observation based attacks against DRL, we propose attacks based on action space and environment dynamics. Among these experiments, we introduce online sequential attack: a novel method specialized for efficiently attacking sequences of temporally consecutive frames, the first targeted attack that perturbs environment dynamics to cause an agent fail in a specific way, and a novel finite difference based black-box attack against DRL. We show empirically that our online sequential attacks can generate effective perturbations in a black-box setting in real time with a small number of queries, independent of episode length. We conduct extensive experiments to compare the effectiveness of different attacks with several baselines in various environments, including game playing, robotics control, and autonomous driving.
This submission sets out to taxonomize evasion-time attacks against deep RL and introduce several new attacks, including two heuristics for efficient evasion-time attacks and attacks that target the environment dynamics and RL system’s actions. The main limitation of this paper is probably its broad scope, which unfortunately prevents it in its current form from addressing each of the goals stated in the introduction systematically to draw conclusive takeaways. 
Taxonomizing the space of adversaries targeting deep RL at test time is a valuable contribution. While the existing taxonomy is a good start, it would be useful if you can clarify the following points in your rebuttal. Why were the “further categorization” items separated from adversarial capabilities? Being constrained to real-time or physical perturbations appears to be another way to describe the adversary’s capabilities. In addition, is there a finer-grained way to characterize the adversary’s knowledge beyond white-box vs. black-box? This binary perspective is common but not very informative. One way to move forward would be for instance to think about the different components of a RL system, and identify those that are relevant to have knowledge of when adversaries are mounting attacks. It would also be helpful to position prior work in the taxonomy. Finally, the taxonomy currently stated in the submissions is more a taxonomy of attacks (or adversaries) than a taxonomy of vulnerabilities, so the title of Section 3 could perhaps be updated accordingly. 
Section 4.1 gives a good overview of different attack strategies against RL based on modifying the observations analyzed by the agent. Many of these attacks are applications of known attack strategies and will be familiar to readers with adversarial ML background (albeit some of these strategies were previously introduced and evaluated against “supervised” classifiers only). One point was unclear however: why is the imitation learning based black-box attack not a transferability-based attack? As far as I could understand, the strategy described corresponds exactly to the commonly adopted strategy of transferring adversarial examples found on a substitute model (see for instance “Intriguing properties of neural networks” by Szegedy et al. and “Practical Black-Box Attacks against Machine Learning” by Papernot et al.). In other words, Section 4.1 could be rescoped to put emphasis on the attack strategies that have not been explored previously in the context of reinforcement learning: e.g., the finite difference approach with adaptive sampling or the universal attack with optimal selection of initial frames. It is unfortunate that the treatment of these two attacks is currently deferred to the appendix as they make the paper more informative. Similarly, Sections 4.2 and 4.3 would benefit from being extended to put forward the new attack threat model considered in these two sections. 
While the introduction claimed to make a systematic evaluation of attacks against RL, the presentation of the experimental section can be improved to ensure the analysis points out the relevant takeaways. For instance, it is unclear what the differences are between results on TORCS and other tasks included in the Appendix. Specifically, results on Enduro do not seem as conclusive as those presented on TORCS. Do you have some intuition as to why that is the case? In Figure 7, it appears that a large number of frames need to be manipulated before a drop on cumulative reward is noticeable. Previous efforts manipulated single frames only, could you stress why the setting is different here? Throughout the section, many Figures are small and it is difficult to infer whether the difference between the white-box and black-box variants of an attack is significant or not. Could you analyze this in more details in the text? In Table 2, how should the L2 distance be interpreted? In other words, when is the adversary successful? 
If you can clarify any of the points made above in your rebuttal, I am of course open to revise my review. 
Editorial details: 
Figures are not readable when printed. 
Figure 5 is improperly referenced in the main body of the paper. 
Figure 7: label is incorrect for Torcs and Hopper (top of figure)