Hindsight Experience Replay
Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task. The video presenting our experiments is available at https://goo.gl/SMrQnI.
Summary:
This paper introduces a method called hindsight experience replay (HER), which is designed to improve performance in sparse reward, RL tasks. The basic idea is to recognize that although a trajectory through the state-space might fail to find a particular goal, we can imagine that the trajectory ended at some other goal state-state. This simple idea when realized via the experience replay buffer of a deep q learning agent improves performance in several interesting simulated arm manipulation domains.
Decision:
I think this paper is an accept. The main idea is conceptually simple (in a good way). The paper is clear, and includes results on a toy domain to explain the idea, and then progresses to show good performance on simulated arm tasks. I have several questions that I invite the authors to respond to, in order to refine my final judgement of the paper.
I will start with high-level questions. The auxiliary task work [1] (and hinted at with the Horde paper), shows that learning to predict and control many reward functions in parallel, results in improvement performance on the main task (for example the score in an atari game). One can view the auxiliary tasks as providing dense reward signals for training a deep network and thus regularizing the agent’s representation of the world to be useful for many task. In the auxiliary task work, pixel control tasks seems to improve performance the most, but the more general idea of feature control was also presented. I think the paper under review should discuss the connections with this work, and investigate what it might mean to use the auxiliary task approach in the simulated arm domain. Are both methods fundamentally getting at the same idea? What are the key differences?
The work under-review is based on episodic tasks with goal states. How should we think about continuing tasks common in continuous control domains? Is the generalization straightforward under the average reward setting?
The related work section mentions prioritized DQN is orthogonal. I don’t agree. Consider the possibility that prioritized DQN performs so well in all the domains you tested, that there remains no room for improvement when adding HER. I think this should be validated empirically.
One could consider options as another approach to dealing with sparse reward domains: options allow the agent to jump through the state space, perhaps making it easier to find the goal state. Clearly there are challenges with discovering options, but the connection is worth mention.
The result on the real robot, is too quickly discussed and doesn’t seem to well support the main ideas of the paper. It feels like: “lets add a robot result to make the paper appear stronger”. I suggest presenting a clearer result or dropping it and pulling in something else from the appendix.
Footnote 3 says count-based exploration method was implemented by first discretizing the state. My understanding of Bellemare’s work was that it was introduced as a generalization of tabular exploration bonuses to the case of general function approximation. Was discretization required here, and if not what effect does it have on performance?
The paper investigates the combination of HER with shaped rewards, showing the shaping actually hurts performance. This is indeed interesting, but a nice companion result would be investigating the combination of HER and shaping in a domain/task where we know shaping alone works well.     
[1] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016). Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397.