Evolved Policy Gradients
We propose a metalearning approach for learning gradient-based reinforcement learning (RL) algorithms. The idea is to evolve a differentiable loss function, such that an agent, which optimizes its policy to minimize this loss, will achieve high rewards. The loss is parametrized via temporal convolutions over the agent's experience. Because this loss is highly flexible in its ability to take into account the agent's history, it enables fast task learning. Empirical results show that our evolved policy gradient algorithm (EPG) achieves faster learning on several randomized environments compared to an off-the-shelf policy gradient method. We also demonstrate that EPG's learned loss can generalize to out-of-distribution test time tasks, and exhibits qualitatively different behavior from other popular metalearning algorithms.
The authors consider a meta learning framework where in the outer loop a parametrized loss function is learned from environment reward with a simple ES approach, and in the inner loop standard policy gradient optimization is performed with the (meta-)learned loss function. The method is demonstrated on two continuous control tasks with body randomization and results show that (during test time) using loss functions learned by the proposed framework, the agent is able to achieve better performance compared to PPO, without observing environment rewards.  The idea of this paper is interesting from multiple perspectives: 1. Meta-learning useful and sharable priors for RL agent from a distribution of randomized tasks. 2. Reducing the reliance of environment rewards during test time (or deployment) 3. How to combine evolution training (ES) with RL, in a hierarchical way.  The use of ES can be better motivated. A question might be: why not using RL with temporal abstraction (such as many Hierarchical RL papers) for the outer loop to learn the parameterized loss? ES may (or may not) be a better approach to RL since the optimization without BP is easier? And it can address longterm credit assignment problem better than RL with temporal abstraction?  The authors claim that one contribution is the learned loss function reduces the reliance of external reward during testing time. This is also a very important message but without much support from experiments or relevant discussions. I think the main problem is the tasks are too simple, thus to me the results in this paper do not imply that this idea could work on real tasks (such sim2real transfer).  This also raises the concern of if the loss function here is really a "loss", in a traditional sense, since the loss function here is actually a parameterized model with additional MLP and temporal convolutions, which is also "transferred" in the evaluation time. This could change the story from learning loss functions to reusing a pretrained hierarchical reward encoding module, which has been well studied before. This also explains why there's no need for external reward using the learned loss function -- the external reward could be already encoded in the additional parameters. One argument could be the evolved loss function can be generalizable to more diverse tasks while simple transfer learning by modulation would fail.  Overall I think the paper has many interesting ideas even though the presentation could be further improved. I recommend acceptance.