HINDSIGHT POLICY GRADIENTS
A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.
This paper extends the work of Hindsight Experience Replay to (goal-conditioned) policy gradient methods. Hindsight, which allows one to learn policies conditioned on some goal g, from off-policy experience generated by following goal g’, is cast in the framework of importance sampling. The authors show how one can simply rewrite the goal-conditioned policy gradient by first sampling a trajectory, conditioned on some goal $g’$ and then computing the closed form gradient in expectation over all goals. This gradient is unbiased if the rewards are off-policy corrected along the generated trajectories. While this naive formulation is found to be unstable , the authors propose a simple normalized importance sampling formulation which appears to work well in practice. To further reduce variance and computational costs, the authors also propose goal subsampling mechanisms, which sample goals which are likely along the generated trajectories. The method is evaluated on the same bit-flipping environment as [1], and a variety of discrete environments (grid worlds, Ms. Pac-Man, simulated robot arm) where the method appears highly effective. Unfortunately for reasons which remain unclear, hindsight policy gradients with value baselines appear unstable.
Quality:
This paper scores high wrt. quality. The theoretical contributions of the method are solid, the experiments are well designed and highlight the efficacy of the method, as well as areas for improvement. In particular, I commend the authors for the rigorous analysis (bootstrapped error estimates, separate seeds for hyper-parameters and reporting test error, etc.), including the additional results found in the appendix (sensitivity and ablative analyses). That being said, the paper could benefit from experiments in the continuous control domain and a direct head-to-head comparison with HER. While I do not anticipate the proposed method to outperform HER in terms of data-efficiency (due to the use of replay) the comparison would still be informative to the reader.
Clarity:
The paper is well written and easy to follow. If anything, the authors could have abridged sections 2 and 3 in favor of other material found in the Appendix, as goal-conditioned policy gradients (and variants) are straightforward generalizations of standard policy gradient methods.
Originality:
Novelty is somewhat low for the paper as Hindsight Experience Replay already presented a very similar off-goal-correction mechanism for actor-critic methods (DDPG). The method is also very similar to [2], the connection to which should also be discussed.
Significance.
Despite the low novelty, I do believe there is value in framing “hindsight” as importance sampling in goal-conditioned policy gradients. This combined with the clear presentation and thorough analysis in my opinion warrants publication and will certainly prove useful to the community. Significance could be improved further should the paper feature a more prominent discussion / comparison to HER, along with a fix for the instabilities which occur when using their method in conjunction with a value baseline.
[1] Hindsight Experience Replay. Marcin Andrychowicz et al.
[2] Data-Efficient Hierarchical Reinforcement Learning. Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine.
Detailed Comments:
* Section 2: “this formulation allows the probability of a state transition given an action to change across time-steps within an episode”. I do not understand this statement, as $p(s_{t+1} \mid s_t, a_t)$ is the same transition distribution found in standard MDPs, and appears stationary wrt. time.
* Theorems 3.1 - 3.1 (and equations). A bit lengthy and superfluous. Consider condensing the material.
* Section 5: I found the change in notation (from lower to upper-case) somewhat jarring. Also, the notation used for empirical samples from the mini-batch is confusing. If $A^{(i)}_t}$ is meant to be the action at time-step $t$ for the $i$-th trajectory in the minibatch, then what does $G^{(i)} = g$ mean ? I realize this means evaluating the probability by setting the goal state to $g$, but this is confusing especially when other probabilities are evaluated conditioned on $G^{(i)}$ directly.
* Section 6. “Which would often require the agent to act after the end of an episode”. Do you mean that most episodes have length T’ < T, and as such we would “waste time” generating longer trajectories ?
* RE: Baseline instabilities. Plotting the loss function for the value function could shed light on the instability.