Published as a conference paper at ICLR 2018 LEARNING AN EMBEDDING SPACE FOR TRANSFERABLE ROBOT SKILLS
We present a method for reinforcement learning of closely related skills that are parameterized via a skill embedding space. We learn such skills by taking advantage of latent variables and exploiting a connection between reinforcement learning and variational inference. The main contribution of our work is an entropyregularized policy gradient formulation for hierarchical policies, and an associated, data-efficient and robust off-policy gradient algorithm based on stochastic value gradients. We demonstrate the effectiveness of our method on several simulated robotic manipulation tasks. We find that our method allows for discovery of multiple solutions and is capable of learning the minimum number of distinct skills that are necessary to solve a given set of tasks. In addition, our results indicate that the hereby proposed technique can interpolate and/or sequence previously learned skills in order to accomplish more complex tasks, even in the presence of sparse rewards.
The paper presents a new approach for hierarchical reinforcement learning which aims at learning a versatile set of skills. The paper uses a variational bound for entropy regularized RL to learn a versatile latent space which represents the skill to execute. The variational bound is used to diversify the learned skills as well as to make the skills identifyable from their state trajectories. The algorithm is tested on a simple point mass task and on simulated robot manipulation tasks.
This is a very intersting paper which is also very well written. I like the presented approach of learning the skill embeddings using the variational lower bound. It represents one of the most principled approches for hierarchical RL. 
Pros: 
- Interesting new approach for hiearchical reinforcement learning that focuses on skill versatility
- The variational lower bound is one of the most principled formulations for hierarchical RL that I have seen so far
- The results are convincing
Cons:
- More comparisons against other DRL algorithms such as TRPO and PPO would be useful
Summary: This is an interesting deep reinforcement learning paper that introduces a new principled framework for learning versatile skills. This is a good paper.
More comments:
- There are several papers that focus on learning versatile skills in the context of movement primitive libraries, see [1],[2],[3]. These papers should be discussed.
[1] Daniel, C.; Neumann, G.; Kroemer, O.; Peters, J. (2016). Hierarchical Relative Entropy Policy Search, Journal of Machine Learning Research (JMLR),
[2] End, F.; Akrour, R.; Peters, J.; Neumann, G. (2017). Layered Direct Policy Search for Learning Hierarchical Skills, Proceedings of the International Conference on Robotics and Automation (ICRA).
[3] Gabriel, A.; Akrour, R.; Peters, J.; Neumann, G. (2017). Empowered Skills, Proceedings of the International Conference on Robotics and Automation (ICRA).