In this paper, a new intrinsic reward generation method for sparse-reward reinforcement learning is proposed based on an ensemble of dynamics models. In the proposed method, the mixture of multiple dynamics models is used to approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model to the mixture of the dynamics models. In order to show the effectiveness of the proposed intrinsic reward generation method, a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization (PPO) algorithm. Numerical results show that for representative locomotion tasks, the proposed model-ensemble-based intrinsic reward generation method outperforms the previous methods based on a single dynamics model.
Summary
This paper proposes a model-based method for intrinsic rewards based on probabilistic neural network ensembles. For a particular ensemble element, an intrinsic reward is defined as a log-term that measures the deviation in prediction between a Gaussian mixture over all ensemble elements and the particular ensemble element (at the previous update period). The intrinsic reward signal applied in practice is the minimum of the aforementioned quantity w.r.t. all ensemble elements. The authors provide some theoretical motivation for their approach and validate their method in sparse-reward continuous robotics domains (Mujoco), using PPO as reference algorithm. Experiments are averaged over 10 seeds and compared against various other intrinsic reward baselines including PPO without intrinsic motivation. In these experiments, the method provided by the authors seems to dominate over competing approaches.
Quality
I find the quality low. First, while I appreciate that the authors try to provide some theoretical motivation for their method, there is quite a gap from theory to practice. The theory assumes that the world model is known and considers a stationary reward signal. In practice, the world model is approximated with the current network ensemble, a quantity that is changing over time yielding a non-stationary signal. Second, the experiments are conducted in a sparse-reward modification of Mujoco-type environments that are non-sparse by construction. Sparsity is introduced by accumulating instantaneous rewards over some time window before "releasing" them. This way of sparsifying rewards yields weird partial observability issues, e.g. the same state-action pair observed at the right moment in time yields significant reward whereas at the wrong moment in time yields no reward at all. Additionally, it is always guaranteed that there will be a reward signal at a certain frequency. There are probably better environments for studying the approach, like Atari for example. I do understand that the authors cite Oh et al. 2018 who apply the same technique of sparsification, but Oh et al. also conduct additional experiments in ALE.
Clarity
The paper is clearly written and easy to follow. On a side note, it could be stated more clearly that the presented approach is not model-based because no forward prediction is required for constructing intrinsic rewards (merely probability values are queried for observed transitions). There is one question I have though regarding the experiments where different intrinsic reward approaches are compared against each other. The paper states that all approaches normalize intrinsic rewards according to Equation (17)---why do all of them then need a different weighting factor \beta as stated in the second paragraph of Section 3.3? Furthermore, since \beta is fine-tuned for each intrinsic reward approach and each environment, can the authors please elaborate in detail how exactly this is accomplished (to ensure correct interpretability of the results)?
Originality
The originality is low. As stated by the authors, the proposed method is an incremental extension of Achiam and Sastry 2017 who proposed a similar method for non-ensemble methods.
Significance
The significance is low as well. The method's improvement over other intrinsic reward approaches is minor (the environments chosen by the authors are also not ideal). I feel the significance is reduced further by the fact that there are other model-based approaches that use models of the proposed kind for increasing sample-efficiency and performance considerably in non-sparse environments (e.g. Kurutach et al.2018).
Update
After reading the other reviews and the rebuttal, the concerns I have raised remain. I still feel this paper shouldn't be accepted to ICLR. However, given the extensive experimental analysis conducted by the authors, I feel a score of 1 from my side was too harsh. I therefore increase to 3.