We present a new approach for efficient exploration which leverages a lowdimensional encoding of the environment learned with a combination of modelbased and model-free objectives. Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional representational space to gauge novelty. We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space. One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy. We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines.
The paper proposes an approach to exploration by utilizing an intrinsic reward based on distances in a learned, evolving abstract representation space. The abstract space is learned utilizing both model-free and model-based losses, and the behaviour policy is based on planning combining the model-free and model-based components with an epsilon-greedy exploration strategy. Learning the abstract representation space itself is based on a previous work, but the contribution of this paper is the utility of it to design the reward bonus for exploration by utilizing distances in this evolving representation space.
As it stands, I am leaning towards rejecting the paper, for the following reasons.
(1) while the idea proposed is interesting, the current work rather explores it in a limited manner which is unsatisfactory.
(2) I think the presentation of the bonus itself -- novelty search (Section 4), which is the core of the paper, is rather unclear. (3) The assumption of deterministic transition dynamics may be ignored in favour of games which seem to be our benchmarks, but the results presented for the control tasks, Table 1, are not statistically significant, and the paper is missing details about the architecture/sweep for the baselines experimented with. 
(4) Parts of the paper is rather unclear/feels disconnected -- for instance, the interpretable abstract representation bit; this was a loss in the original work, and seems to be just mentioned arbitrarily here while the loss isn't really used (unless it is used, and not mentioned in the paper).
(5) Overall, the proposed reward bonus is a heuristic whose specific design choice isn't statistically shown to be useful (Ablation in Appendix), and the empirical results comparing to other methods are underwhelming.
Here are my main points of concern which I hope the authors address in the rebuttal:
(1) Designing reward bonuses to induce exploratory behaviour in the agent has seen a surge of publications in the Deep RL literature in recent years. The key property all these methods aim for is a bonus that pushes the agent to the boundaries of its current "known region", and then rely on the stochasticity due to epsilon-greedy to cross that boundary -- pushing this boundary further. While this is different from exploration to reduce uncertainty, it is nonetheless a reasonable approach leading to competitive policies when evaluated in deep RL. But a characteristic all these bonuses aim for is that they fade away with time -- for instance count-based bonus are inversely proportional to visit counts, or prediction error bonuses go to 0 as the prediction becomes more accurate. But what do these novelty bonuses converge to? Is it just a stationary value based on consecutive loss parameter (in which case the hope is they don't affect the external reward scale, they just shift it uniformly)?
(2) What exactly are the nearest neighbours? Is it a search based on the data in the buffer or is it a notion of temporal neighbours?
(3) If it's temporal, why would there ever be biased for some states -- "We do this in order..novel states".
(4) I was completely unable to understand the section in the Appendix which is making a case for the ranked weighting. If you have a succinct explanation for the heuristic it'd be great.
(5) Further, as a heuristic it is mentioned that l2 norm may not be effective if the dimensionality of the representation space is increased. So why the heuristic? I think it either needs more empirical validation, or a theoretical justification.
(6) While the evaluation scheme used in the paper to quantify the exploration of the behaviour policy is interesting -- y-axis of plots in Figure 4 for the Labyrinth task -- why/what exactly is the role of Figure 2? Is the interpretability loss used here? Is it to reason for utilizing e-greedy instead of a purely-greedy behaviour? I think this is a little unclear, and can be better clarified. Further, the distinction of primary and secondary features is interesting, but their clear demarcation is rather questionable in more complicated domains -- in the abstract space.
(7) Do you have a hypothesis for why the 1-step value functions are not sufficient for decision making in this simple domain -- labyrinth - with the abstract representations?
(8) If model-based algorithms get more steps to learn shouldn't model-free too? I'm not sure I understand the reasoning for the experiment design choice.
(9) Whats the architecture used for Bootstrap DQN? It needs to have multiple heads -- but based on the current architecture that doesn't seem likely.
(10) Are the extrinsic rewards ignored in learning -- "only focus on intrinsic rewards" (Section 6.2.2)? If they are for the proposed method, are they for the competitors too? If so why, and what is the reward for Bootstrap DQN?
(11) I think the Discussion section raises interesting points about interpretability and metric learning, but I do think the conclusions drawn are a little inflated.
(12) The ablation study in Section D of the Appendix is not statistically significant -- so why is wighted reward useful? Please comment.
(13) How would stochasticity in transition dynamics affect the abstract representation space? Discussing this would be very interesting.
(14) Learning curves for the control tasks?
Comments about typos/possible points of confusion:
(1) The last para in Section 6.1 -- discusses "open" labyrinth heat map, then what do we mean by learning the dynamics of the wall? There is no wall in open, right?
(2) In Section 4 -- I think x_{t+1} is an estimate from the unrolled model -- \hat{x}_{t+1}? Further, it would be helpful to mention that it is an estimate based on the learned model.
(3) n_freq is used in the pseudocode in the main paper -- but no mention of it to explain it is made in the main.
(4) Contrasting the work to existing literature would be useful (in the Related Work section; as opposed to summarizing existing work).
(5) buffered Q network --> target networks?