Meta-Reinforcement Learning of Structured Exploration Strategies
Exploration is a fundamental challenge in reinforcement learning (RL). Many current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we study how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -model agnostic exploration with structured noise (MAESN) -to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.
This paper introduces a metalearning approach that discovers RL exploration strategies that work well across a particular distribution of MDPs. Furthermore, due to the metalearning approach used, the exploration behaviors can be adapted to the problem at hand quickly as the agent gains experience in the MDP.  It is shown that this approach significantly outperforms other metalearning strategies such as MAML (which metalearns initial policy parameters), as well as random exploration.  This appears to be due to the fact that the proposed method conditions on a per-episode latent variable that encourages temporally coherent exploration throughout the episode, rather than per-step randomness.  Strengths:  + The core idea of per-episode temporally-consistent exploration is excellent and much needed in the RL community.  Per-step random search will never scale well to complex problems.  This is an idea that will be useful to the community well beyond this particular instantiation of it.  + Most exploration techniques are just glorified random or uniform exploration (e.g. intrinsic motivation, diversity metrics, etc). Instead, this approach uses the learned latent space to discover a space of useful exploratory behaviors.  + Metalearning seems to be the right tool for this.  The technical development of the MAESN algorithm is both sound and insightful, as is the analysis of why approaches that are limited to time-invariant noise don’t work.  + The paper is written extremely clearly, being both technically thorough, and providing good high-level overviews of tricky concepts before diving it.  + The evaluation is quite thorough as well.  It is shown that the MAESN works significantly better than existing state-of-the-art approaches on a number of robot domains and that MAESN leads to sensible, temporally consistent exploration. There is also a solid intuitive explanation given of the results + analysis.  + I liked that early “learning to learn” papers were cited in addition to more recent metalearning papers.  Too often metalearning is talked about as though it is a brand new idea, so it is good to see some older works getting credit.   Weaknesses:  - No significant weaknesses.  There could have been a larger diversity in domains (e.g. non-robotics domains such as scheduling, etc.).    Other thoughts:  - There may be something I’m missing, but I’m wondering why the KL divergence term was used instead of just clamping the pre-update w_i to the prior, since they converge to that anyway.  Does it just make it harder to learn if there isn’t some flexibility there during learning?  - There are a few other related papers that would be good to cite.  This work is related in spirit to some prior work on searching for reward functions that speed up learning on a distribution of MDPs by essentially encouraging certain types of exploration.  Some examples:  @article{niekum2010genetic,   title={Genetic programming for reward function search},   author={Niekum, Scott and Barto, Andrew G and Spector, Lee},   journal={IEEE Transactions on Autonomous Mental Development},   volume={2},   number={2},   pages={83--90},   year={2010},   publisher={IEEE} }  @inproceedings{sorg2010internal,   title={Internal rewards mitigate agent boundedness},   author={Sorg, Jonathan and Singh, Satinder P and Lewis, Richard L},   booktitle={Proceedings of the 27th international conference on machine learning (ICML-10)},   pages={1007--1014},   year={2010} }