We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies. We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies. Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies. This framework for training deep RL policies involve a zero-sum dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region. Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm. Without loss of generality, we demonstrate that deep RL policies trained in this fashion will be maximally robust to a "worst" possible adversarial disturbances.
There are two anonymity violations in the paper. The first is in the sentence "The 7-DoF robot result presented shortly previously appeared in our abstract that introduced robust GPS [21]". The second is in the first linked video, which links to a non-anonymized youtube video. The second linked video, a dropbox link, does not have the correct permissions set, and thus cannot be viewed. Also, the citation style does not seem to follow the ICLR style guidelines.
Disregarding the anonymity and style violations, I will review the paper.  I do not have background in H_inf control theory, but I will review the paper to the best of my ability.
This paper proposes a guided policy search method for training deep neural network policies that are robust to worst-case additive disturbances. To my knowledge, the approach presented in the paper is novel, though some relevant references are missing. The experimental results demonstrate the method on two simulated experimental domains, demonstrating robustness to adversarial perturbations. The paper is generally well-written, but has some bugs and typos. The paper is substantially longer than the strongly suggested page limit. There are parts of the paper that should be moved to an appendix to accommodate the page limit.
Regarding the experiments:
My main concerns are with regard to the completeness of the experiments. First, the experimental results report performance in terms of cost/reward, which is extremely difficult to interpret. It would be helpful to also provide success rate for all experiments, where the authors can define success as, e.g. getting the peg in the hole or being within a certain threshold of the goal.
Second, the paper should provide a comparison of policy robustness between the proposed approach and (1) a policy trained with standard GPS, (2) a policy trained with GPS and random perturbations, and ideally, (3) prior approaches to robustness, e.g. Pinto et al., Madelkar et al. [1], or Rajeswaran et al. [2].
Regarding related work and clarity:
There are a few papers that consider the problem of building deep neural network policies that are robust [1,2] that should be discussed and ideally compared to.
Recent deep reinforcement learning work has studied the problem of robustness to adversarial perturbations in the observation space, e.g. [3,4,5,6]. As such, it would be helpful to clarify in the introduction that this paper is considering additive perturbations in the action space.
The paper switches between using rewards and costs. It would be helpful to pick one term and stick with it for the entire paper, rather than switching. Further, it seems like there are errors due to the switching. e.g. on page 3, \ell is defined as the expected reward and in equation 3, it seems like the protaganist policy is trying to minimize \ell, contradicting the earlier definition.
Lastly, section 5.1 is currently rather difficult to follow. It would help to see more top-down direction in the derivation and more details in section 5.1, 5.2, and 5.3 to be moved to an appendix.
Regarding correctness:
There seem to be some issues in the math and/or notation:
The most major issue is in Algorithm 2, which is probably the most important part of the paper to be correct, given that it provides a complete picture of the algorithm. I believe that steps 3 and 4 are incorrect and/or incomplete. Step 4 should be referring to the local policy p rather than the global policy pi (assuming the notation in sections 5.1 and 5.3). Also, the local policy p(v|x) appears nowhere in the algorithm, and probably should appear in step 4. In step 5, how is this regression different from step 7?
In equation 1 and elsewhere in section 4, there is a mix of notation, using u and z inter-changably. (e.g. in equation 1 and the following equation, I believe the u should be switched to z or the z should be switched to u).
Minor feedback:
> "requires fewer samples to generalize to the real world"
None of these experiments are in the real world, so the term "real world" should not be used.
> "algoruithm" -> "algorithm"
> "when unexpected such as when there exists" - bad grammar / typo
> "ertwhile sensy-sensitivity parameter" - typo
- reference 1 is missing authors
In summary, I think the paper would be significantly improved with further experimental comparisons, further discussion of related work, and clarifications/corrections on the notation, equations, and algorithms.  In its current form, I don't think the paper is fit for publication. My rating is between a 4 and a 5.
[1] http://vision.stanford.edu/pdf/mandlekar2017iros.pdf
[2] https://arxiv.org/abs/1610.01283
[2] https://arxiv.org/abs/1701.04143
[3] https://arxiv.org/abs/1702.02284
[4] https://www.ijcai.org/proceedings/2017/0525.pdf
[5] https://arxiv.org/abs/1705.06452