While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an ongoing challenge. In this work, we propose an effective reward shaping method through predictive coding to tackle sparse reward problems. By learning predictive representations offline and using these representations for reward shaping, we gain access to reward signals that understand the structure and dynamics of the environment. In particular, our method achieves better learning by providing reward signals that 1) understand environment dynamics 2) emphasize on features most useful for learning 3) resist noise in learned representations through reward accumulation. We demonstrate the usefulness of this approach in different domains ranging from robotic manipulation to navigation, and we show that reward signals produced through predictive coding are as effective for learning as hand-crafted rewards.
*Synopsis*:
  This paper proposes using the features learned through Contrastive Predictive Coding as a means for reward shaping. Specifically, they propose to cluster the embedding using the clusters to provide feedback to the agent by applying a positive reward when the agent enters the goal cluster. In more complex domains they add another negative distance term of the embedding of the current state and goal state. Finally, they provide empirical evidence of their algorithm working in toy domains (such as four rooms and U-maze) as well as a set of control environments including AntMaze and Pendulum.
  Main Contributions:
  - Using the embedding learned through Contrastive Predictive Coding for reward shaping.
  - A reward shaping scheme that seems generally applicable to any embedding.
  *Review*:
  I think the paper provides a compelling motivation, and is well written. I think using the embedding learned through CPC could provide meaningful semantics for representation learning and for reward shaping (as done in the current paper), and encourage the authors to continue down this line of inquiry. Unfortunately, I have several concerns over the method as currently implemented and the empirical comparisons (specifically with the chosen competitors) which I detail below. Given these concerns I am unwilling to recommend accepting this paper, unless several of these concerns are addressed.
  1. This algorithm, by nature, is purely offline as the CPC and clustering all are currently done offline. Furthermore, the clustering portion of this approach requires states to be randomly sampled from the environment to create a nice set of clusters which are representative of the environment's full state space. These two requirements significantly limit this approach, especially when looking at domains where simulation is not possible, or the underlying state distribution is unknown. By and all, I don't think this means we should completely discount this method entirely and the authors do mention this as a detriment to their algorithm in section 6.3. I'm wondering if this paper should look at implementing an online version before publication, but think this is less prescient to the other concerns.
  2. I am concerned about the current policy learning scheme (i.e. tiered policy (1) go to the correct cluster (2) go to the goal) which seems to only be used by your approach. This invalidates the comparisons made with the other reward shaping schemes as this goes beyond reward shaping (you are learning two separate policies).
  3. The current competitors are unsatisfactory as they don't include other reward shaping techniques from the literature. Also the related works section seems to completely disregard this part of the literature. I would recommend comparing your approach to (methods you even cite!):
     - "Reward Shaping via Meta Learning" by Zou et. al https://arxiv.org/pdf/1901.09330.pdf
     - "Potential based reward shaping" by Gao et. al. https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/viewPaper/10930
     (I'm sure there are others beyond these)
  4. I'm also concerned with the current significance of the results, particularly AntMaze, Half Cheetah, and Reacher. I'm especially concerned because I don't feel your competitors are not relevant/representative of the current state of reward shaping.
  5. It would be worthwhile to try this method on several RL algorithms (i.e. Q-learning, TRPO, SAC, DDPG, etc...). This will help readers understand if this approach is a general method, or only applicable to PPO.
  6. I quite like the idea of predictive coding (albeit the original scheme presented by Rao and Ballard 1999) as an unsupervised representation learning scheme, but am unsure this is critical for your method and the current approach is not really predictive coding in a sense (or at least the ideas I'm familiar with from cognitive computational neuroscience). I am concerned with the predictive coding idea being highlighted here as a key ingredient, but none of the papers containing the originating idea of predictive coding are mentioned. Rao and Ballard is one, but there is a rich literature following from this work into the free energy formulation (Friston) and active learning. These ideas should appear in your introduction, as you heavily rely on them. Also there are other predictive coding schemes for unsupervised representation learning (such as PredNets from David Cox https://coxlab.github.io/prednet/), which I believe should be mentioned. In light of these other methods, I'm not sure your discussion on only using predictive features for reward shaping is accurate, and instead these claims should be softened for only features learned through CPC.
  Missing experimental settings:
  - Number of runs tested
  - What are the error bars in your plots?
  I would also like to recommend "Deep Reinforcement Learning that Matters" by Henderson for a reference on how to conduct meaningful Deep RL experiments using policy gradient methods (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewPaper/16669). I think this paper could benefit from the recommendations made there.
  More questions/clarifications:
  
  Q1: What constitutes success in the grid world domains for table 1?
  Q2: If you were to train a single policy using your reward shaping scheme (i.e. not the tiered approach currently employed) how would the new policy perform? Are there problems with the current scheme where you have to have the tiered policy?
  Q3: Why CPC and not say an autoencoder or VAE? A comparison over many types of unsupervised embedding learning algorithms could be interesting and make your method more general than currently presented. It could also strengthen your argument for using CPC.
  Q4: How long were the trajectories for training CPC?
  *Other minor comments not taken into account in the review*
  - I think your labels are backwards in table 1.