DHER: HINDSIGHT EXPERIENCE REPLAY FOR DYNAMIC GOALS
Dealing with sparse rewards is one of the most important challenges in reinforcement learning (RL), especially when a goal is dynamic (e.g., to grasp a moving object). Hindsight experience replay (HER) has been shown an effective solution to handling sparse rewards with fixed goals. However, it does not account for dynamic goals in its vanilla form and, as a result, even degrades the performance of existing off-policy RL algorithms when the goal is changing over time. In this paper, we present Dynamic Hindsight Experience Replay (DHER), a novel approach for tasks with dynamic goals in the presence of sparse rewards. DHER automatically assembles successful experiences from two relevant failures and can be used to enhance an arbitrary off-policy RL algorithm when the tasks' goals are dynamic. We evaluate DHER on tasks of robotic manipulation and moving object tracking, and transfer the polices from simulation to physical robots. Extensive comparison and ablation studies demonstrate the superiority of our approach, showing that DHER is a crucial ingredient to enable RL to solve tasks with dynamic goals in manipulation and grid world domains.
In this paper, the authors extend the HER framework to deal with dynamical goals, i.e. goals that change over time.
In order to do so, they first need to learn a model of the dynamics of the goal, and then to select in the replay buffer experience reaching the expected value of the goal at the expected time. Empirical results are based on three (or four, see the appendix) experiments with a Mujoco UR10 simulated environment, and one experiment is successfully transfered to a real robot.
Overall, the addressed problem is relevant (the question being how can you efficiently replay experience when the goal is dynamical?), the idea is original and the approach looks sound, but seems to suffer from a fundamental flaw (see below).
Despite some merits, the paper mainly suffers from the fact that the implementation of the approach described above is not explained clearly at all.
Among other things, after reading the paper twice, it is still unclear to me:
- how the agent learns of the goal motion (what substrate for such learning, what architecture, how many repetitions of the goal trajectory, how accurate is the learned model...)
- how the output of this model is taken as input to infer the desired values of the goal in the future: shall the agent address the goal at the next time step or later in time, how does it search in practice in its replay buffer, etc.
These unclarities are partly due to unsufficient structuring of the "methodology" section of the paper, but also to unsufficient mastery of scientific english. At many points it is not easy to get what the authors mean, and the paper would definitely benefit from the help of an experienced scientific writer.
Note that Figure 1 helps getting the overall idea, but another Figure showing an architecture diagram with the main model variables would help further.
In Figures 3a and 5, we can see that performance decreases. The explanation of the authors just before 4.3.1 seem to imply that there is a fundamental flaw in the algorithm, as this may happen with any other experiment. This is an important weakness of the approach.
To me, Section 4.5 about transfer to a real robot does not bring much, as the authors did nothing specific to favor this transfer. They just tried and it happens that it works, but I would like to see a discussion why it works, or that the authors show me with an ablation study that if they change something in their approach, it does not work any more.
In Section 4.6, the fact that DHER can outperform HER+ is weird: how can a learn model do better that a model given by hand, unless that given model is wrong? This needs further investigation and discussion.
In more details, a few further remarks:
In related work, twice: you should not replace an accurate enumeration of papers with "and so on".
p3: In contrary, => By contrast, 
which is the same to => same as
compare the above with the static goals => please rephrase
In Algorithm 1, line 26: this is not the algorithm A that you optimize, this is its critic network.
line 15: you search for a trajectory that matches the desired goal. Do you take the first that matches? Do you take all that match, and select the "best" one? If yes, what is the criterion for being the best?
p5: we find such two failed => two such failed
that borrows from the Ej => please rephrase
we assign certain rules to the goals so that they accordingly move => very unclear. What rules? Specified how? Please give a formal description.
For defining the reward, you use s_{t+1} and g_{t+1}, why not s_t and g_t?
p6: the same cell as the food at a certain time step. Which time step? How do you choose?
The caption of Fig. 6 needs to be improved to be contratsed with Fig. 7.
p8: the performance of DQN and DHER is closed => close?
DHER quickly acheive(s)
Because the law...environment. => This is not a sentence.
Mentioning in the appendix a further experiment (dy-sliding) which is not described in the paper is of little use.