Published as a conference paper at ICLR 2018 CONTINUOUS ADAPTATION VIA META-LEARNING IN NONSTATIONARY AND COMPETITIVE ENVIRONMENTS
The ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.
---- Summary ----
This paper addresses the problem of learning to operate in non-stationary environments, represented as a Markov chain of distinct tasks. The goal is to meta-learn updates that are optimal with respect to transitions between pairs of tasks, allowing for few-shot execution time adaptation that does not degrade as the environment diverges ever further from the training time task set.
During learning, an inner loop iterates iterates over consecutive task pairs. For each pair, (T_i, T_{i+1}) trajectories sampled from T_i are used to construct a local policy that is then used to sample trajectories from T_{i+1}. By calculating the outer-loop policy gradient with respect to expectations of the trajectories sampled from T_i, and the trajectories sampled from T_{i+1} using the locally optimal inner-loop policy, the approach learns updates that are optimal with respect to the Markovian transitions between pairs of consecutive tasks.
The training time optimization algorithm requires multiple passes through a given sequence of tasks. Since this is not feasible at execution time, the trajectories calculated while solving task T_i are used to calculate updates for task T_{i+1} and these updates are importance weighted w.r.t the sampled trajectories' expectation under the final training-time policy.
The approach is evaluated on a pair of tasks. In the locomotion task, a six legged agent has to adapt to deal with an increasing inhibition to a pair of its legs. In the new RoboSumo task, agents have to adapt to effectively compete with increasingly competent components, that have been trained for longer periods of time via self-play.
It is clear that, in the locomotion task, the meta learning strategy maintains performance much more consistently than approaches that adapt through PPO-tracking, or implicitly by maintaining state in the RL^2 approach. This behaviour is less visible in the RoboSumo task (Fig 5.) but it does seem to present. Further experiments show that when the adaptation approaches are forced to fight against each other in 100 round iterated adaptation games, the meta learning strategy is dominant. However, the authors also do point out that this behaviour is highly dependent on the number of episodes allowed in each game, and when the agent can accumulate a large amount of evidence in a given environment the meta learning approach falls behind adaptation through tracking. The bias that allows the agent to learn effectively from few examples precludes it from effectively using many examples.
---- Questions for author ----
Updates are performed from \theta to \phi_{i+1} rather than from \phi_i to \phi_{i+1}. Footnote 2 states that this was due to empirical observations of instability but it also necessitates the importance weight correction during execution time. I would like to know how the authors expect the sample in Eqn 9 to behave in much longer running scenarios, when \pi_{\phi} starts to diverge drastically from \pi_{\theta} but very few trajectories are available.
The spider-spider results in Fig. 6 do not support the argument that meta learning is better than PPO tracking in the few-shot regime. Do you have any idea of why this is?
---- Nits ----
There is a slight muddiness of notation around the use of \tau in lines 7 & 9 in  of Algorithm 1. I think it should be edited to line up with the definition given in Eqn. 8. 
The figures in this paper depend excessively and unnecessarily on color. They should be made more printer, and colorblind, friendly.
---- Conclusion ----
I think this paper would be a very worthy contribution to ICLR. Learning to adapt on the basis of few observations is an important prerequisite for real world agents, and this paper presents a reasonable approach backed up by a suite of informative evaluations. The quality of the writing is high, and the contributions are significant. However, this topic is very much outside of my realm of expertise and I am unfamiliar with the related work, so I am assigning my review a low confidence.