We provide a framework for incorporating robustness -to perturbations in the transition dynamics which we refer to as model misspecification -into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.
AFTER REBUTTAL:
The authors answered all my questions and (significantly) modified the paper according to all the comments from all reviewers. The paper has significant value given the extensive experimental results and I believe could provide insight to other researchers in the field. I still have (like other reviewers) some concerns about scalability regarding the space of possible perturbations, but, as it is, the paper is worth for publication. For these reasons, *I increase my score to 7*, since I believe it's better than a weak accept (ignore the system rating, since it doesn't allow to put 7). 
======================================================
Summary:
The authors provide a framework to incorporate robustness against perturbed dynamics to RL agents. They build upon the MPO algorithm and incorporate a few modifications that enable such robustness and entropy regularization. They show in multiple experiments that the robust versions of MPO outperform the non-robust ones when testing on environments with novel perturbations.
Decision:
I am happy with the experiments conducted on this paper and I think they would be a nice contribution to the community. However, I have serious concerns about the novelty of the proposed approach. The authors claim many novel contributions, but most of them are special cases of existing literature (not cited), or are simple modifications to existing approaches. For this reason my score is a reject. If the authors clearly cite the existing literature, state their contributions accordingly, and redirect the paper more towards their good experimental results (which they have plenty) I would be willing to substantially upgrade my score. Find below my supporting arguments.
About novelty
=============
The authors claim to have a novel framework and novel MDP formulations  for robust MDPs disregarding a substantial volume of the literature on robustness in the context of planning, control, reinforcement learning and MDPs. Since the authors really consider two types of robustness (robustness to dynamics and robustness to action stochasticity) I will expose in the following the relevant literature that is missing on both types, and when combining both.
Robustness to Dynamics:
Bart van den Broek, Wim Wiegerinck, and Hilbert J. Kappen.  Risk sensitive path integral control. In UAI, 2010.
Arnab Nilim and Laurent El Ghaoui.   Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005
Wolfram Wiesemann, Daniel Kuhn, and Berc  Rustem.  Robust markov decision processes. Mathematics of Operations Research, 38(1):153–183, 2013.
Lars Peter Hansen and Thomas J Sargent.Robustness. Princeton university press, 2008
Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7):1298–1328, 2014.
Yinlam  Chow,  Aviv  Tamar,  Shie  Mannor,  and  Marco  Pavone.   Risk-sensitive  and  robust decision-making: a cvar optimization approach.   In Advances in Neural Information Pro-cessing Systems, pages 1522–1530, 2015.
Robustness to Action Stochasticity:
Roy Fox, Ari Pakman, and Naftali Tishby.  G-learning: Taming the noise in reinforcement learning via soft updates.arXiv preprint arXiv:1512.08562, 2015.
Jonathan Rubin, Ohad Shamir, and Naftali Tishby.  Trading value and information in mdps.In Decision Making with Imperfect Decision Makers, pages 57–74. Springer, 2012.
Daniel A Braun, Pedro A Ortega, Evangelos Theodorou, and Stefan Schaal.  Path integral control  and bounded  rationality.   In Adaptive  Dynamic  Programming  And  ReinforcementLearning (ADPRL), 2011 IEEE Symposium on, pages 202–209. IEEE, 2011.
Combination of both:
[1] Grau-Moya, Jordi, et al. "Planning with information-processing constraints and model uncertainty in Markov decision processes." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2016.
Importantly, in [1] above, it is shown a very similar line of research as the authors are proposing. In [1] it is combined entropy regularization with robustness in dynamics. In particular, using the formulation in [1] one can see how the formulations in the present paper can be recovered by setting the parameters in various ways. For example,
1- letting the entropy regularization coefficient ($\alpha$ in [1]) to a finite value and the robustness coefficient ($\beta$ in [1]) to $-\infty$ one recovers the traditional Robust MDP formulation.
2- Selecting a uniform prior over discrete theta (where each component in theta would correspond to a particular perturbation)  and setting $\beta \rightarrow 0$ one can also recover Soft-RE formulation (although a non-MPO version of it, i.e. just the MDP formulation).
3- Similarly, one can get rid of the entropy regularization by setting $\alpha \rightarrow \infty$.
Note that in [1] the uncertainty set is determined by a KL constraint whereas here it is determined by the chosen perturbations. However, in [1] setting the proper prior and $\beta$, one can obtain the same behaviour.
Importantly, in all previous references multiple contraction proofs (for entropy regularization Bellman operators, for robust operators alone, and for robust and entropy regularized operators) have already been discovered. 
I hope all of the previous points can convince the authors about my decision above regarding novelty. Note, please, that I am aware that a robust version of MPO is slightly novel, however, a robust MDP formulation is more general than that and it could have perfectly been applied to any other RL optimization algorithm. Therefore, I don't think adding robustness to MPO alone is sufficient for enough novelty (other wise we would publish 1 paper per method converting one by one all known algorithms into their robust version of it). 
About scalability
=================
I liked the experimental section. However, I have a question about practicality of the approach. As the authors show in the paper, the proposed method scales well in terms of  high-dimensional state and action spaces. But what about the scalability in terms of possible perturbations? I can imagine a plausible situation where we are dealing with a high-dimensional robot, let's say a humanoid robot, which allows for many possible "perturbations" e.g. longer left leg, longer right leg, longer fingers, shorter thumbs, and so on. The space of possible perturbations can be potentially very big. Could the authors elaborate on scalability issues of this type? Have they already thought about how one would solve this? I imagine that perturbing, let's say in one dimension, might change the dynamics quite in a different way compared to perturbing in another dimension. It would be interesting to see how their method generalizes when testing on unseen "dimension" perturbations.
Although this might present problems in terms of generalization, I understand that it does not invalidate the results presented here.
Good experimental results:
I have carefully checked all experimental results and I must say that they are very well executed e.g. multiple ablation studies and sufficient environments to support their approach.  Maybe a small comment: In figures 6 and 8 in the appendix there are multiple plots that only differ in the data (the titles are the same, e.g. two hopper figures with the same title). Maybe the authors can correct for this.