In this paper, we advocate the use of explicit memory for efficient exploration in reinforcement learning. This memory records structured trajectories that have led to interesting states in the past, and can be used by the agent to revisit those states more effectively. In high-dimensional decision making problems, where deep reinforcement learning is considered crucial, our approach provides a simple, transparent and effective way that can be naturally combined with complex, deep learning models. We show how such explicit memory may be used to enhance existing exploration algorithms such as intrinsically motivated ones and count-based ones, and demonstrate our method's advantages in various simulated environments.
In this paper, the authors propose an exploration strategy based on the explicit storage and recall of trajectories leading to novel states. A pool of such trajectories is managed over time, and a method is proposed so that the agent can learn how to follow a path corresponding to these trajectories so as to explore novel states. The idea is demonstrated in a set of room experiments, and quickly shown efficient in Montezuma's Revenge and PrivateEye Atari games.
Overall, the idea has some merits, but the empirical study is weak and the paper suffers from unsufficient writing effort (or more probably time).
What I like most in the paper is the split of exploration methods into 3 categories: adding some "intrinsic reward" bonuses to novel states (curiosity-driven exploration) , trying to reach various goals (goal-driven exploration) and using memory to reach again novel states (memory-driven exploration). Actually, this split may be debated. For instance, some frameworks based on goals have been labelled curiosity-driven, e.g. "Curiosity-Driven Exploration of Learned Disentangled Goal Spaces" (Laversanne-Finot, Péré and Oudeyer, CoRL 2018), but anyways I find it useful. That said, this aspect of the introduction is reiterated in the "Related Work" section in a quite redundant way, whereas both parts could have been better integrated. Furthermore, the related work section is hardly a draft, I'll come to that later.
The presentation of the method in Section 2 is rather clear and convincing. My only concern is about the assumption that the agent is always starting in the same state. This assumption may not hold in many settings, and the approach appears to be quite dependent on it. A discussion of how it could be extended to a less restricting assumption would be welcome.
The experimental section is weaker. A few concerns:
- I could not find much about the number of seeds, trials, the way to depict some variance, the statistical significance of the differences between results presented in Figure 1. The same is true about Figs. 2, 3 and 5.
- In Fig.2, the claim that the author's method learns better models is hardly supported by the left-hand-side plot, and significance is not assessed.
- I'm puzzled about the very low performance of baselines in the plots of Fig. 3. Could the author explain why these performances are null.
- The Atari games section helps figuring out that the framework is not too specific of the rooms environment, but the lack of analysis does not help making sure that this is just the explicit recall mechanism that is responsible for superior performance and why.
Another point about this section is that poor writing does not help understanding some points.
- to me, the first sentence of Section 3.2.2 does not make sense at all.
- in the caption of Fig. 4, "The second row is the heatmaps for states that the number of times being selected as a target state.": I don't get what it means, thus I don't understand what that row shows.
- Fig.5 comes with no caption
About the related work:
- The comparison to other methods using memory needs to be expanded. In particular, I would put HER-like mechanisms here rather than in 4.1, as "explicit recall" shares some importan ideas with "experience replay"
- Section 4.4 (HRL) is not useful as is.
Finally, in the conclusion, the claim that the method can be combined with "many sota exploration methods" is not supported, as the authors have only tried two and did not analyse the results in much details.
typos:
- p4:
we can easily counting
(include borders) => including
is provide => provided
are less less-visited states: quite inelegant
- p7:
In Montezuma's Revenge, Comparing => comparing
Where they encourage => remove "Where"
- p8:
recallcan => recall can
the problem of reach goals => reaching
it succesfully reproduce => reproduces
The last paragraph of Section 4.2 needs a careful rewriting, as long sentences with parenthese in the middle appear to be some draft version.
control(Pritzel => Missing space
Our method use memory => uses
Although ..., but => remove but
The path function can be seen as a form of skills => skill?
Besides, the "can be seen" needs to be further explained...
Appendix
Finally, we provided => provide
is around (math formula) => cannot you be more specific?