Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
Efficient exploration remains a challenging research problem in reinforcement learning, especially when an environment contains large state spaces, deceptive or sparse rewards. To tackle this problem, we present a diversity-driven approach for exploration, which can be easily combined with both off-and on-policy reinforcement learning algorithms. We show that by simply adding a distance measure regularization to the loss function, the proposed methodology significantly enhances an agent's exploratory behavior, and thus prevents the policy from being trapped in local optima. We further propose an adaptive scaling strategy to enhance the performance. We demonstrate the effectiveness of our method in huge 2D gridworlds and a variety of benchmark environments, including Atari 2600 and MuJoCo. Experimental results validate that our method outperforms baseline approaches in most tasks in terms of mean scores and exploration efficiency.
[Summary]: The paper proposes a formulation for incentivizing efficient exploration in deep reinforcement learning by encouraging the diversity of policies being learned. The exploration bonus is defined as the distance (e.g., KL-divergence) between the current policy and a set of previous policies. This bonus is then added to the standard reinforcement learning loss, whether off-policy or on-policy. The paper further uses an adaptive scheme for scaling the exploration bonus with respect to external reward. The first scheme, as inspired from [Plappert et.al. 2018], depends on the magnitude of exploration bonus. The second scheme, on the other hand, depends on the performance of policy with respect to getting the external reward.  [Paper Strengths]: The paper is clearly written with a good amount of details and is easy to follow. The proposed approach is intuitive and relates closely to the existing literature.  [Paper Weaknesses and Clarifications]: - My major concerns are with the experimental setup:     (a) The paper bears a similarity in various implementation details to Pappert et.al. [5] (e.g. adaptive scaling etc.), but it chose to compare with the noisy network paper [8]. I understand [5] and [8] are very similar, but the comparison to [5] is preferred, especially because of details like adaptive scaling etc.     (b) The labels in Figure-5 mention that DDPG w/ parameter noise: is this method from Plappert et.al. [5] or Fortunato et.al. [8]. It is unclear.     (c) No citations are present for the names of baseline methods in the Section-4.3 and 4.4. It makes it very hard to understand which method is being compared to, and the reader has to really dig it out.     (d) Again in Figure-5, what is "DDPG(OU noise)"? I am guessing its vanilla DDPG. Hence, I am surprised as to why is "DDPG (w/ parameter space noise)" is performing so much worse than vanilla DDPG? This makes me feel that there might be a potential issue with the baseline implementation. It would be great if the authors could share their perspective on this.     (e) I myself compared the plots from Figure-1,2,3, in Pappert et.al. [5] to the plots in Figure-5 in this paper. It seems that DDPG (w/ parameter space noise) is performing quite worse than their TRPO+noise implementation. Their TRPO+noise beats the vanilla TRPO, but DDPG+noise seems to be worse than DDPG itself. Please clarify the setup.     (f) Most of the experiments in Figure-4 seems to be not working at all with A2C. It would be great if authors could share their insight. - On the conceptual note: In the paper, the proposed approach of encouraging diversity of policy has been linked to "novelty search" literature from genetic programming. However, I think that taking bonus as KL-divergence of current policy and past policy is much closer to perturbing policy with a parameter space noise. Both the methods encourage the change in policy function itself, rather than changing the output of policy. I think this point is crucial to the understanding of the proposed bonus formulation and should be properly discussed. - Typo in Line-95  [Final Recommendation]: I request the authors to address the clarifications and comments raised above. My current rating is marginally below the acceptance threshold, but my final rating will depend heavily on the rebuttal and the clarifications for the above questions.  [Post Rebuttal] Reviewers have provided a good rebuttal. However, the paper still needs a lot of work in the final version. I have updated my rating.