LEARNING SELF-IMITATING DIVERSE POLICIES
The success of popular algorithms for deep reinforcement learning, such as policygradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difficultly in credit assignment. Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, but have been found to suffer from high sample complexity by completing forgoing the temporal nature of the problem. Improving the efficiency of RL algorithms in real-world problems with sparse or episodic rewards is therefore a pressing need. In this work, we introduce a self-imitation learning algorithm that exploits and explores well in the sparse and episodic reward settings. We view each policy as a state-action visitation distribution and formulate policy optimization as a divergence minimization problem. We show that with Jensen-Shannon divergence, this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from experience replays. Experimental results indicate that our algorithm works comparable to existing algorithms in environments with dense rewards, and significantly better in environments with sparse and episodic rewards. We then discuss limitations of self-imitation learning, and propose to solve them by using Stein variational policy gradient descent with the Jensen-Shannon kernel to learn multiple diverse policies. We demonstrate its effectiveness on a challenging variant of continuous-control MuJoCo locomotion tasks.
The paper describes a method to improve reinforcement learning for task with sparse rewards signals.
The basic idea is to select the best episodes from the system's experience, and learn to imitate them step by step as the system evolves, aiming at providing a less sparse learning signal.
The math works out to a gradient that is of similar form as a policy gradient, which makes it easy to interpolate both of them. The resulting training procedure is a policy gradient that gets additional reinforcement of the system's best runs.
The experiments show the validity especially for the most extreme case (episodic rewards), while, as expected, for the other extreme of dense rewards, the method's effect is not consistently positive.
The paper then critiques its own method and identifies a critical weakness: the reliance on good exploration. I like that a lot. The paper goes on to suggest an extension to address this by training an ensemble, and shows the effectiveness of this for a number of tasks. However, I feel that the description of this extension is less clear than that of the core idea, and introduces too many new ideas and concepts in a too condensed text.
The paper seems a significant in that it provides a notable improvement for sparse-rewards tasks, which are a common sub-class of real-world problems.
My background is not RL. While I am quite confident in my understanding of the paper's math, I am not 100% familiar with the typical benchmark sets. Hence, I cannot judge whether the results include good baselines, or whether the task selection is biased. I can also not judge the completeness of the related work, and how novel the work is. For these questions, I hope that the other reviewers can provide more information.
Pros:
 - intuitive idea for a common problem
 - solution elegantly has the form of a modified policy gradient
 - convincing experimental results
 - self-critique of core idea, and extension to address its main weakness
 - nicely written text, does not leave a lot of questions
Cons:
 - while the core idea is nicely motivated and described and good to follow, Section 2.3 feels very dense and too short.
Overall, I find the core idea quite intuitive and elegant. The paper's background, motivation, and core method are well-written and, with some effort, quite readable for someone who is not an RL expert. I found that several questions I had during reading were preempted promptly and addressed. However, the description of the secondary method (Section 2.3) is too dense.
To me, the paper solidly meets the threshold of publication. Since I have no good comparison to other papers, I rate it a "clear accept" (8).
Minor points:
I noticed a few superfluous "the", please double-check.
In Table 1, please use the same exponent for directly comparable numbers, e.g. instead of "1.8e5 4.4e4", say "18e4 4.4e4". Or best just print the full numbers without exponent, I think you have the space.
When reading Table 1, I could bnot immediately line up "PPO" and "Self-imitation" in the caption with the table columns. It took a while to infer that PPO refers to \nu=0, and SI to \nu=0.8. Can you add PPO and SI to the table headings?
You define p as "the masking probability", but it is not clear whether that is the probability for keeping a "1" in the mask,
or for masking out the value. I can only guess from the results. I suggest to rephrase as "the probability of retaining a reward". Also, how about using plain words in Table 1's heading, such as "Noisy rewards\nSuppressing 10% of rewards", so that one can understand the table without having to search for its description in the text?