PROMP: PROXIMAL META-POLICY SEARCH
Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during metatraining as well as ineffective task identification strategies. This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL. Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients. By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning. Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.
In this paper, the authors investigate the gradient calculation in the original MAML (Finn et al. 2017) and E-MAML (Al-Shedivat et al. 2018). By comparing the differences in the gradients of these two algorithms, the authors demonstrate the advantages of the original MAML in taking the casual dependence into account. To obtain the correct estimation of the gradient through auto-differentiation, the authors exploit the DiCE formulation. Considering the variance in the DiCE objective formulation, the authors finally propose an objective which leads to low-variance but biased gradient. The authors verify the proposed methods in meta-RL tasks and achieves comparable performances to MAML and E-MAML. 
Although the ultimate algorithm proposed by this paper is not far away from MAML and E-MAML, they did a quite good job in clarify the differences in the existing variants of MAML from the gradient computation perspective and reveal the potential error due to the auto-differentiation. The proposed new objective and the surrogate is well-motivated from such observation and the trade-off between variance and bias. 
My major concern is how big the effect is if we use (3) comparing to (4) in calculate the gradient. As the authors showed, the only difference between (3) and (4) is the weights in front of the term \nabla_\theta\log\pi_\theta: the E-MAML is a fixed weight and the MAML is using a adaptive through the inner product. Whether the final difference in Figure 4 between MAML and E-MAML is all caused by such difference in gradient estimation is not clearly. In fact, based on the other large-scale high-dimension empirical experiments in Figure 2, it seems the difference in gradient estimator (3) and (4) does not induced too much difference in final performances between MAML and E-MAML. Based on such observation, I was wondering the consistent better performance of the proposed algorithm might not because the corrected gradient computation from the proposed objective. It might because the clip operation or other components in the algorithm. To make a more convincing argument, it will be better if the authors can evaluate different gradient within the same updates.
I am willing to raise my score if the author can address the question. 
minor:
The gradients calculation in Eq (2) and (3) are not consistent with the Algorithm and the appendix.
The notation is not consistent with common usage: \nabla^2 is actually used for denoting the Laplace operator, i.e., \nabla^2 = \nabla \cdot \nable, which is a scalar.