Online Robust Policy Learning in the Presence of Unknown Adversaries
The growing prospect of deep reinforcement learning (DRL) being used in cyberphysical systems has raised concerns around safety and robustness of autonomous agents. Recent work on generating adversarial attacks have shown that it is computationally feasible for a bad actor to fool a DRL policy into behaving sub optimally. Although certain adversarial attacks with specific attack models have been addressed, most studies are only interested in off-line optimization in the data space (e.g., example fitting, distillation). This paper introduces a Meta-Learned Advantage Hierarchy (MLAH) framework that is attack model-agnostic and more suited to reinforcement learning, via handling the attacks in the decision space (as opposed to data space) and directly mitigating learned bias introduced by the adversary. In MLAH, we learn separate sub-policies (nominal and adversarial) in an online manner, as guided by a supervisory master agent that detects the presence of the adversary by leveraging the advantage function for the sub-policies. We demonstrate that the proposed algorithm enables policy learning with significantly lower bias as compared to the state-of-the-art policy learning approaches even in the presence of heavy state information attacks. We present algorithm analysis and simulation results using popular OpenAI Gym environments.
This submission introduces a novel framework for handling adversarial attacks on the observation space used by RL value estimation techniques. The new method is particularly interesting because it claims to address a few issues with current methods: the proposed method is attack-model agnostic, online, and adaptive. Although I find the work very interesting and I consider the problems that the paper is studying very important, I am not very familiar with most of the work that the authors have cited, in part because this work builds on many ArXiv pre-prints that I was not aware of.  I find this paper to be very hard to read for someone who is not familiar with the latest results in addressing adversarial attacks on Deep RL. The authors chose some weird notational standards, the algorithmic  framework is carelessly presented, and the main theoretical result is barely interpreted or compared to previous work. Although I appreciate the empirical results and the claimed improvements, I don't think this paper is very accessible.   Please see a list of specific comments.   L25, 55: is there a reason why some citations are listed separately? I.e. why [9], [10] and not [9, 10]? L28-31: Can you add some references to support the first statement in the Introduction? Particularly, the phrase "even small perturbations in state inputs can result in significant loss ..." seems like something that should have been quantified or empirically demonstrated already. L34: "the the" -> "the" L43: What does it mean to "optimize the observation space" ? Please fix this ambiguity. Table 1: It seems odd to add the column on "Mitigation", as all the listed methods support it. Also, what is the difference between a "robust method" and "a method supporting mitigation"?  L49: Please define the TRPO acronym (e.g. on L27) L55, L85, L174: You should avoid using the term "state-of-the-art" in isolation. Both the context (i.e. environment type, assumptions) and the comparison metric should always be clear whenever a method is considered state-of-the-art. L65: "in [20] in" -> "in [20] for" Section 2: The MDP notation is very sloppy here. E.g.  * It is unclear whether you assume deterministic policies,  * Are we dealing with episodic tasks? Is T fixed, probabilistic?  * The convention is most of the literature is to use "E_{\pi}" to define the return. Any particular reason why the more cumbersome "E_{s_0,a_0, ...}" was chosen here? Figure 1: Is there a reason you chose not to reference the ICLR 2018 paper? Also, why didn't you use the same illustration as they used, as the alternative one you provided does not seem more informative? It seems good to maintain consistency with the work you cite. L104 (eq (3)): I have hard time reading this expectation. What is the index i used for and what does it mean to condition on "m_i, s_i"? Please specify exactly the process for which we compute the return in Eq (3).  Section 3, first paragraph. I really like the motivation provided here, especially the use of the advantage function. Consider adding a sentence on the fact that the discussion applies mostly in the case where we have good estimates for the Q-value and not in the early stages of learning.  L148: "can not" -> "cannot" L154 (eq (5)): Just like Eq (3), this equation is overloaded and hard to interpret. First, how can an action be equal to a policy? I think you should replace "\pi_{*,t}" with "\pi_{*} (s_t)". Also, I am confused by the last part of the equation, when a_{master,t} is said to be part of the set "\{\pi_{nom}, \pi_{adv}}". What does that even mean? Please clarify. Figure (2): I believe it would be useful to explain what "m_i"'s correspond to in the proposed architecture. L164: "reported [13]" -> "reported in [13]". L215: Isn't the "actual advantage" the one defined under the true advantage? Consider using "observed advantage" instead.  L223: I find it odd that you refer to [11] here instead of the paper that derived those bounds: Kakade & Langford (2002). Moreover, I believe you should add a more extensive discussion here on how one should interpret Proposition 1 and 2 + comparison to Kakade & Langford's bounds and a proper comparison is granted here. L267: I think it would be enough to say here "s_i + U(-a,a) for a>0" instead of the convoluted presentation in the submitted paper.   L275: "Openai" -> OpenAI  ----After author feedback---- I would like to thank the authors for the additional feedback, which provides clarifications for the major conceptual concerns I have raised. Moving some important clarifying remarks from the Supplemental material to the main paper is a great suggestion. Also, I agree that improving the accessibility of the paper to a wider audience is necessary and will increase the value of this work.