Training reinforcement learning policies in partially observable domains with sparse reward signal is an important and open problem for the research community. In this paper, we introduce a new sequence-level intrinsic novelty model to tackle the challenge of training reinforcement learning policies in sparse rewarded partially observable domains. First, we propose a new reasoning paradigm to infer the novelty for the partially observable states, which is built upon forward dynamics prediction. Different from conventional approaches that perform self-prediction or one-step forward prediction, our proposed approach engages open-loop multi-step prediction, which enables the difficulty of novelty prediction to flexibly scale and thus results in high-quality novelty scores. Second, we propose a novel dual-LSTM architecture to facilitate the sequence-level reasoning over the partially observable state space. Our proposed architecture efficiently synthesizes information from an observation sequence and an action sequence to derive meaningful latent representations for inferring the novelty for states. To evaluate the efficiency of our proposed approach, we conduct extensive experiments on several challenging 3D navigation tasks from ViZDoom and DeepMind Lab. We also present results on two hard-exploration domains from Atari 2600 series in Appendix to demonstrate our proposed approach could generalize beyond partially observable navigation tasks. Overall, the experiment results reveal that our proposed intrinsic novelty model could outperform several state-of-the-art curiosity baselines with considerable significance in the testified domains.
This paper extends the prediction-error based model by Pathak et. al., 2019 by learning a forward (and inverse) dynamics model for predicting a state feature multiple steps into the future (say, K-steps) given an open loop sequence of K actions as opposed to 1 step into the future, with the caveat that instead of using learnable state features, a random network is used for computing state features, similar to Random Network Distillation (RND) by Burda et. al., 2019. Also, their inverse dynamics models predicts the entire sequence of actions up to K steps. Experiments on VizDoom point-navigation tasks show that the proposed model does better than baselines as rewards get sparser. Ablations are provided to justify the choice of K in multi-step prediction, the choice of inverse dynamics and the choice of RND state features.
My decision is weak reject as:
1. The paper does a good job at clearly explaining their model, presenting results on relevant experiments and baseline comparisons for their model and justifying each modeling choice with ablations.
2. The novelty in their contribution is moderate - the idea of long-term future prediction is not new (e.g.: Ke et. al., 2019) (but using it for giving a curiosity bonus is new), the architecture choice is not significantly new.
3. I expected a more detailed ablation for the choice of K, given that “multi-step” has major emphasis in the paper, but Figure 7(a) only shows ablations for 3 vs 1 step predictions.
Elaborating on (3):
- My major concern is that the gap between 1-step prediction and 3-step prediction (in Figure 7 (a)) is not significant. Note that the version of their model with 1-step predictions does not completely reduce to Pathak et. al. 2019’s ICM model, as RND state features are used. I feel that it may be the case that the 1-step version of the proposed model is actually good enough to beat all the baselines and adding multiple steps gives marginal gains. This hypothesis needs to be verified by the authors with more experiments - I would like to see all the main experiments have an additional baseline of 1-step predictions. 
- Only two values of K are tested - 1 and 3, what happens with larger values of K?
Other comments:
The motivation for using long-term predictions to “infer more meaningful novelty” is fine on it’s own but seems to conflict with the choice of random network (RND) state features. Random features imply that a random “hash” of the observations is being computed which has no reason to have similar features for two nearby states. If there is any small amount of noise in the state transitions, this would mean that predicting far into the future is practically impossible given that the random feature of slightly incorrect states will be very different. Can the authors give reasons/motivations as to why such a model would work in the case of stochastic transitions and K is large or will it be brittle to stochasticity?
References:
All references are same as those cited in paper.