This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproducing past good trajectories, which can potentially make long-term credit assignment easier when rewards are delayed. GASIL can be easily combined with any policy gradient objective by using GASIL as a learned reward shaping function. Our experimental results show that GASIL improves the performance of proximal policy optimization on 2D Point Mass and MuJoCo environments with delayed reward and stochastic dynamics.
[Paper Summary]:
This paper proposes a regularization technique for existing RL algorithms by encouraging them to learn to reproduce the best past trajectories which obtained higher reward than that of current policy. The proposed method in the paper has the same high-level idea as "Self-imitation learning" [Oh et.al. ICML 2018] with a different objective. Instead of performing imitation learning to distill the knowledge from past best trajectories, this proposes to use inverse reinforcement learning via GAIL objective [Ho and Ermano, 2016]. The best k trajectories from past experience are stored to train a discriminator which is then used to augment the external reward function with a discriminator reward.
[Paper Strengths]:
The paper combines ideas from GAIL and self-imitation learning to propose a method that leverages past best trajectories via inverse-RL. This combination allows one to interpret self-imitation of best trajectories as a mechanism for "reward shaping" where learned discriminator shapes the environmental reward using past experiences. This is an exciting perspective and needs further discussion.
[Paper Weaknesses and Clarifications]:
=> This paper is very closely related to self-imitation learning [Oh et.al.], however, there is no theoretical justification provided (unlike [Oh et. al.]) whether the policy learned by optimizing Equation-11 is in anyway related to the optimal policy -- which was the case as shown in [Oh et.al.]. That being said, this is not a requirement for a paper to show theoretical justification as long as the paper justifies given approach with ample empirical evidence.
=> The main comparison point for the proposed approach, "GASIL", is "SIL" [Oh et.al.]. This paper provides a good comparison on continuous control tasks on Mujoco where "GASIL" performs slightly better than "SIL" in 3 out of 6 environments. However, "SIL" [Oh et. al.] showed extensive experiments on all Atari Games + Mujoco tasks. Since the proposed approach is mainly empirically motivated, the experiments should at least show a comparison on all the environments of the closely-related prior work. It would be much more convincing to see a bar chart across all 48 Atari Games showing relative improvement of "GASIL" over "SIL", as shown in Figure-4 of [Oh et. al.].
=> Other concerns:
    - The paper mentions on multiple occasions that the proposed method would handle delayed and "sparse" reward. However, it is not clear how can past best trajectories help with "sparse" rewards ("delayed-dense-rewards" seems alright, but they are not the same as "sparse"!). For instance, suppose the agent gets only terminal-reward in a maze. In such a case, the agent would need to rely on some form of exploration bonus (count-based, curiosity etc.) to reach the sparse-goal even once.
    - What prevents the learned policy from over-fitting to the local minima of the "locally" best trajectories seen so far?
[Final Recommendation]:
I request the authors to address the comments raised above. The paper has good potential, but sufficient empirical evidence is needed to justify the proposed technique. If the results on all Atari games can be included and shown to improve over "SIL", I would update my final rating.