A major challenge in reinforcement learning is exploration, especially when reward landscapes are sparse. Several recent methods provide an intrinsic motivation to explore by directly encouraging agents to seek novel states. A potential disadvantage of pure state novelty-seeking behavior is that unknown states are treated equally regardless of their potential for future reward. In this paper, we propose an exploration objective using the temporal difference error experienced on extrinsic rewards as a secondary reward signal for exploration in deep reinforcement learning. Our objective yields novelty-seeking in the absence of extrinsic reward, while accelerating exploration of reward-relevant states in sparse (but nonzero) reward landscapes. We implement the objective with a two-policy Q-learning method in which Q and Q x are the action-value functions for extrinsic and secondary rewards, respectively. Secondary reward is given by the absolute value of the TD-error of Q. Training is off-policy, based on a replay buffer containing a mix of trajectories sampled using Q and Q x . We characterize performance on a set of continuous control benchmark tasks, and demonstrate comparable or faster convergence on all tasks when compared with other state-of-the-art exploration methods.
This paper proposes an exploration algorithm for reinforcement learning agents based on learning a separate Q-value function which treats TD-errors as rewards. Experiments on continuous control tasks with a difficult exploration component are used to highlight the effectiveness of the approach. 
The proposed algorithm is interesting and has an intuitive appeal, the magnitude of the TD-error as an "auxiliary reward" seems like a natural choice to guide exploration. The experiments are comprehensive, including some ablation studies and robustness tests, with one caveat. Due to some experimental details (particularly concerning the environments), the empirical results may be invalid and it is difficult to assess them. There are also some certain parts that are unclear in the experimental results. Overall, because of these issues, I cannot recommend acceptance although I would be willing to increase my score if my concerns are addressed.
Concerning the SparseHalfCheetah task, from Appendix F: "...baseline reward level is -1 while a successful state provides 0 reward, but report reward values on a 0 to 500 scale for direct comparison with previous work". This reward function is not sparse since it gives -1 at every step except at the goal which would incentivize the agent to explore naturally if it were acting greedily (or approximately greedily) with respect to the value function. For a sparse reward task, I would expect the reward to be 0 everywhere except at the goal state(s), where it would have some positive value. This is the case for the original reward function for SparseHalfCheetah described in [1]. This change to the reward function makes any comparisons to previous work questionable (Table 1) and could also affect the qualitative results in section 4.5. 
Similarly, the other 3 tasks also do not have a sparse reward structure as the agent receives -1 at every timestep. This again is a confounder for assessing the exploration capabilities of the algorithm. It would be preferable to use other environments such as those in [1], which indeed have sparse rewards. 
For these sparse reward problems where most rewards are zero, then before any nonzero reward is observed, it seems like Qx would mostly behave like previous algorithms based on a state novelty term (as discussed in sec 3.4). This could happen since the agent would only be observing rewards of 0 at this point during training. It is unclear to me then which benefits QXplore could have compared to previous algorithms such as DORA.
Other points:
1) Some discussion of possible pitfalls for the algorithm could be added. For example, while the noisy TV problem may be avoided, there could be a noisy reward problem now. The absolute TD error could be high simply due to stochasticity in rewards. The agent could also be vulnerable to scenarios in which a stochastic transition brings the agent to either a state eventually leading to very high reward or to a state leading to no reward as this would cause the TD error be high (even if the value function converges).
2) The sentence before sec 3.3, "Further TD-error-based exploration with a dedicated exploration policy removes the exploitation-versus-exploration
tradeoff that ... to an optimal Q-function." Could the authors clarify how TD-error-based exploration avoids the exploitation-versus-exploration
tradeoff? I do not see the connection here.
3) In the experiments, how are actions chosen for the epsilon-greedy baseline since the action space is continuous (and not discrete)? 
4) I am bit confused as to what the difference between the lines Q and Qx are in Fig. 3.
5) Why was DORA not included as a baseline algorithm? It would seem to be the most closely related to QXplore when rewards are mostly constant (or zero).
6) Are two separate buffers necessary instead of a single shared buffer (with uniform sampling)? It seems to needlessly introduce complexity and additional hyperparameters (the ratios of samples from each buffer). If this is a key choice, then it should be mentioned in the paper.
The text could also be clarified to indicate there are two buffers. In some places the writing suggests there is only one, e.g. sec3.3 "...policy with a replay buffer shared between Q_\theta andthe Q-function maximizing rx, which we term Qx."
7) In section 4.4, for the "Single-Policy QXplore", why was Q(s,a) replaced by V(s)? If I understand correctly, this experiment tries to test a variant where the TD error is used as a reward bonus. If this is the case, it seems like the best comparison would be to leave the original Q(s,a) which is learned by Q-learning instead of changing it to a state-value function. 
Suggestions (did not impact score)
- I wonder if there is a connection between Qx and the variance of the returns since the latter can be learned by using the squared TD-errror as a reward (see the Direct Variance algorithm in [2]) and the absolute TD-error is a similar quantity. In this way, it could be possible to frame QXplore as following a type of risk-seeking stategy (see [3] for an algorithm that makes use of the variance). 
- I am not sure 'adversarial' is the right term to describe Q and Qx since the two policies are not in direct competition with each other.
[1] "VIME: Variational Information Maximizing Exploration" by Houthooft et al.
[2] "Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods" by Sherstan et al. 
[3] "Deep Reinforcement Learning with Risk-Seeking Exploration" by Dilokthanakul and Shanahan