Control policies, trained using the Deep Reinforcement Learning, have been recently shown to be vulnerable to adversarial attacks introducing even very small perturbations to the policy input. The attacks proposed so far have been designed using heuristics, and build on existing adversarial example crafting techniques used to dupe classifiers in supervised learning. In contrast, this paper investigates the problem of devising optimal attacks, depending on a well-defined attacker's objective, e.g., to minimize the main agent average reward. When the policy and the system dynamics, as well as rewards, are known to the attacker, a scenario referred to as a white-box attack, designing optimal attacks amounts to solving a Markov Decision Process. For what we call black-box attacks, where neither the policy nor the system is known, optimal attacks can be trained using Reinforcement Learning techniques. Through numerical experiments, we demonstrate the efficiency of our attacks compared to existing attacks (usually based on Gradient methods). We further quantify the potential impact of attacks and establish its connection to the smoothness of the policy under attack. Smooth policies are naturally less prone to attacks (this explains why Lipschitz policies, with respect to the state, are more resilient). Finally, we show that from the main agent perspective, the system uncertainties and the attacker can be modelled as a Partially Observable Markov Decision Process. We actually demonstrate that using Reinforcement Learning techniques tailored to POMDP (e.g. using Recurrent Neural Networks) leads to more resilient policies.
This paper investigates the design of adversarial policies (where the action of the adversarial agent corresponds to a perturbation in the state perceived by the primary agent). In particular it focuses on the problem of learning so-called optimal adversarial policies, using reinforcement learning.
I am perplexed by this paper for a few reasons:
1)	What is the real motivation for this work?  The intro argues “casting optimal attacks is crucial when assessing the robustness of RL policies, since ideally, the agent should learn and apply policies that resist *any* possible attack”.   If the goal is to have agents that are robust to *any* attacks, then they cannot be robust just to so-called optimal attacks.  And so what is really the use of learning so-called optimal attacks?
2)	The notion itself of “optimal” attack is not clear.  The paper does not properly discuss this.  It quickly proposes one possible definition (p.4): “the adversary wishes to minimize the agent’s average cumulative reward”.  This is indeed an interesting setting, and happens to have been studied extensively in game-theoretic multi-agent systems, but the paper does not make much connection with that literature (apart from a brief mention at bottom of p.2 / top of p.3), so it’s not clear what is new here compared to this.   It’s also not discussed whether it would ever be worthwhile considering other notions of optimality for the adversary, and what would be the properties of those.
So overall, while I find the general area of this work to be potentially interesting, the current framing is not well motivated enough, and not sufficiently differentiated from other work in robust MDPs and multi-agent RL to make a strong contribution yet.
More minor comments:
-	P.3: “very different setting where the adversary has a direct impact on the system” => Clarify what are the implications of this in terms of framework, theory, algorithm.
-	P.4: You assume a valid Euclidean distance for the perturbed state.  Is this valid in most MDP benchmarks?  How is this implemented for the domains in the experiments?  What is the action space considered? Do you always assume a continuous action space for the attacker?
-	P.5: “we can simply not maintain distributions over actions” -> Why not?  Given the definition of perturbation, this seems feasible.
-	P.5:  Eqn 4 is defined for a very specific adversarial reward function. Did you consider others?  Is the gradient always easy to derive?
-	P.6: Eqn (5) & (6): What is “R” here?
-	P.7: Figure 1, top right plot. Seems here that the loss is above 0 for small \epsilon.  Is this surprising?  Actually improving the policy?
-	P.7: What happens if you consider even greater \epsilon?  I assume the loss is greater.  But then the perturbation would be more detectable?  How do you think about balancing those 2 requirements of adversarial attacks?  How should we formalize detectability in this setting?
-	Fig.2: Bottom plots are too small to read.
-	Sec.6:  Can you compare to multi-agent baselines, e.g. Morimoto & Doya 2005.
-	P.8: “We also show that Lipschitz policies have desirable robustness properties.” Can you be more specific about where this is shown formally?  Or are you extrapolating from the fact that discrete mountain car suffers more loss than continuous mountain car?  I would suggest making that claim more carefully.