Exploration has been a long standing problem in both model-based and model-free learning methods for sensorimotor control. While there has been major advances over the years, most of these successes have been demonstrated in either video games or simulation environments. This is primarily because the rewards (even the intrinsic ones) are non-differentiable since they are function of the environment (which is a black-box). In this paper, we focus on the policy optimization aspect of the intrinsic reward function. Specifically, by using a local approximation, we formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization -much like supervised learning instead of reinforcement learning. This leads to a significantly sample efficient exploration policy. Our experiments clearly show that our approach outperforms both on-policy and off-policy optimization approaches like REINFORCE and DQN respectively. But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module. See project videos at https: //doubleblindICLR.github.io/robot-exploration/.
This paper presents an interesting way to reformulate intrinsic curiosity as a differentiable function. The authors compare the differentiable function against using prediction error via REINFORCE and DQN, showing that their intrinsic curiosity method results in more interactions with unseen objects than the other two methods. For DQN this is to be expected, but it shows that backprop through this function is more efficient than reinforce in getting to unseen state spaces. I think this is an interesting method/proposal and is a somewhat novel reformulation of intrinsic error, but I do have some concerns in comparisons/claims. 
In the introduction, the authors say that the intrinsic curiosity method proposed by Pathak et al. is sample inefficient and isn’t tested in robots. However, to my understanding the REINFORCE baseline isn’t really equivalent (though it may be possible that it is, it was unclear how exactly the loss was formulated in the baseline, did include the other components from Pathak et al.?). If the claim is that this method is more efficient, I think it should have compared against that method directly. 
Moreover, I think the description of the experiments doesn’t provide enough information. For example, the method says that different learning rates were used for the min-max game to stabilize it, but doesn’t say what they were. 
Also, for the DQN baseline what were the parameters? Was there an epsilon greedy policy on top of the exploration reward? Was this annealed as in other work? Generally, I think more detail is needed throughout (even if it just refers to a more detailed appendix).
Overall, I think this work needs to be revised to include more details on hyperaparameters, details on the baselines, and describing differences between Pathak et al.’s method and the REINFORCE baseline. Moreover, feedback from other comments on this work should be addressed which reflect in more detail my comments below on opinionated claims (e.g., https://openreview.net/forum?id=SkzeJ3A9F7&noteId=HJlFlZOa2X )
Comments/Thoughts:
+ I think in the introduction there are some statements that probably need citations. For example, “But the same formulation from an optimization viewpoint, it suffers from all the bad properties of extrinsic rewards. The reward is a function of environment behavior with respect to the performed action. Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefficient.” —> Why is this true? Is there a citation that can back this? Do you prove it later in the paper? 
+ “Yes, 54 environments but no real-world physical robots” —> this and the intro seems like a blogpost at times. That can be fine (some would argue it’s a good thing), but there seem to be some opinions without citations/backing, I suggest trying to back up statements wherever possible and avoid opinions. For example in this statement, robots aren't a requirement for evaluating intrinsic motivation.
+ “Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefficient.” —> citation/backing? it might be nice to point to the experiment section here to back it (e.g., "As will be shown in Section X and in \citet{something}, REINFORCE can be quite sample inefficient")
+ “In practice, the existing on-policy algorithms, e.g., A3C (Mnih et al., 2016), PPO (Schulman et al., 2017) etc. are deployed off-the shelf -> This is confusing, so is this using REINFORCE or PPO/A3C? what is this statement referring to?
+ “regress to rti to learn value estimates (i.e., off-policy) as discussed in the previous section” —> regress to \sum r_t{I} for a value estimate?? Value is the expected return so not sure if this is a typo or i missed something earlier
+ What is the actual loss function used for the baseline? Is it the same as Pathak et al.?
+ What are the hyper parameters for DQN exploration? What are all the hyper parameters for any/all the algorithms? 
+ Was a variance-reducing baseline used in REINFORCE?
+ What is the variance representing in the graphs, std across several trials? Maybe I missed it, but how many trials represent this standard deviation?
+ “Hence, we train the forward predictor slightly faster than the policy by keeping higher learning rate to stabilize the learning process. “ —> what were the learning rates?
Linguistic/Typos:
Also, some minor, but frequent, grammatical issues/typos that I’ve added below could be fixed. I would ask that the authors please have the submission proof-read for English style and grammar issues. There are many minor mistakes, some of which I’ve tried to point out below. 
+ “This leads to a significantly sample efficient exploration policy. “ —> significantly more (?) sample efficient ?
“Why is that? To understand the reason behind sample inefficiency of curiosity or intrinsic rewards, notice how the intrinsic rewards are given by agent” —> by the agent?
“Forward model fθF is trained to minimize its loss which amounts to minimizing rti with respect to θF” —> the forward model
“However, policy is optimized to maximize the objective” —> However, the policy
“We can also optimize  for policy parameters θP via differentiable loss function” —> We can also optimize for (the) policy parameters \theta via (a) differentiable loss function?
“To optimize policy to maximize a discounted sum “ —> To optimize the policy
“How good is Forward Prediction Model” —> How good is the forward prediction model
There are several other spots, but basically another pass over the paper might be worth it to check for these sorts of issues.