Policy optimization struggles when the reward feedback signal is very sparse and essentially becomes a random search algorithm until the agent accidentally stumbles upon a rewarding or the goal state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or more ad-hoc measures of surprise. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploration based on forward prediction in the representation space. Our experiments show the state of the art performance on challenging locomotion task with continuous control and on image-based exploration tasks with discrete actions on Atari.
The paper proposes an approach for exploration via reward bonuses based on a form of surprise. The surprise factor is based on the next state of a particular transition, and the error in the embedding space to satisfy a linear dynamics formulation. The embedding space of the states and actions are optimized to increase the mutual-information in predicting next state, and current action - encouraging meaningful embeddings with more training, and hence gradual fading away of the extrinsic rewards.
The paper is mostly well-written, and the idea is interesting. The experimental results do show that the proposed reward augmentation leads to better performing policies, but the claims in the experimental section need to be less strong ("outperforms the baseline by a large margin" - Figure 4 - overlapping error bars; "state of the art" - Figure 5 - again, error bars, and no improvement in some domains.) But overall, I think the paper can be accepted as it is an interesting approach.
Below are some comments that I hope the authors address in their rebuttal, followed by some possible typos in the current draft.
- Theorem 1 content placement: The organization here is rather unclear. Currently, you present Theorem 1, and then talk about using "JSD instead of MI". Maybe this is a last minute mistake. In either case, it is strongly suggested that the section be reworked to be clearer.
- JSD is upper bounded by ln(2); your bounds (7) and (8) would change consequently too.
- Training regime employed:
  3 epochs-512 minibatch -> assuming distinct minibatches are sampled, (512x3) samples used
  Collected (5k*500; sparsehalfcheetah)/(50*500; swimmercatcher)/(100k*4500; atari)
  Is this the sample usage for training? Axis labels for all plots are missing -- specifically scale of x-axis.
  Commenting on the sample complexity -- especially as the embedding network seems easy-to-train (or insufficiently trained), would be good; optimizing a lower-bound insufficiently leads one to doubt if the bound is meaningful at all. Is the huge batch of samples mostly used in TRPO/RL part of the infrastructure?
- Discussing extrinsic rewards: the pros. vs. cons of the two reward formulations, why both are used etc. would be useful.
- Embedding dimension: d=8 in Gravitar and Solaris, but performance is less significant (no significance) in these domains. Is this due to insufficient training?
- RL method: Including details about form of TRPO used in appendix would be good (vine/single-path). Further if entropy regularization is used, how does the exploration interplay work.
- A.2 is an interesting section. A linear dynamics model being effective in MuJoCo tasks seems plausible. But an Atari example is definitely more interesting. Therefore this section can be clearer - specifically distinction between residual error and sample error.
Typos:
- Appendix \lambda parameters unclear
- Appendix step-size information contradictory.
Post-response comment: while I do think the approach is interesting, the utility of it is mostly demonstrated currently through empirical experiments. These experiments are preliminary, but used to make strong arguments for the effectiveness of the proposed approach. Further, upon highlighting this in my review, the authors disagree and think it’s empirical validity is rather superior. This leaves me concerned, and after thinking about it further, I do not think this is sufficient for acceptance. Therefore, I’m reducing my score to a 5.
PS: the characterization of irreducible error as a product of the limitation of a linear model may be inaccurate.