Temporal Difference learning with function approximation has been widely used recently and has led to several successful results. However, compared with the original tabular-based methods, one major drawback of temporal difference learning with neural networks and other function approximators is that they tend to over-generalize across temporally successive states, resulting in slow convergence and even instability. In this work, we propose a novel TD learning method, Hadamard product Regularized TD (HR-TD), to reduce over-generalization and thus leads to faster convergence. This approach can be easily applied to both linear and nonlinear function approximators. HR-TD is evaluated on several linear and nonlinear benchmark domains, where we show improvement in learning behavior and performance.
This paper introduces a variation on temporal difference learning for the function approximation case that attempts to resolve the issue of over-generalization across temporally-successive states. The new approach is applied to both linear and non-linear function approximation, and for prediction and control problems. The algorithmic contribution is demonstrated with a suite of experiments in classic benchmark control domains (Mountain Car and Acrobot), and in Pong.
This paper should be rejected because (1) the algorithm is not well justified either by theory or practice, (2) the paper never clearly demonstrates the existence of problem they are trying to solve (nor differentiates it from the usual problem of generalizing well), (3) the experiments are difficult to understand, missing many details, and generally do not support a significant contribution, and (4) the paper is imprecise and unpolished.
Main argument
The paper does not do a great job of demonstrating that the problem it is trying to solve is a real thing. There is no experiment in this paper that clearly shows how this temporal generalization problem is different from the need to generalize well with function approximation. The paper points to references to establish the existence of the problem, but for example the Durugkar and Stone paper is a workshop paper and the conference version of that paper was rejected from ICLR 2018 and the reviewers highlighted serious issues with the paper—that is not work to build upon. Further the paper under review here claims this problem is most pressing in the non-linear case, but the analysis in section 4.1 is for the linear case. 
The resultant algorithm does not seem well justified, and has a different fixed point than TD, but there is no discussion of this other than section 4.4, which does not make clear statements about the correctness of the algorithm or what it converges to. Can you provide a proof or any kind of evidence that the proposed approach is sound, or how it’s fixed point relates to TD?
The experiments do not provide convincing evidence of the correctness of the proposed approach or its utility compared to existing approaches. There are so many missing details it is difficult to draw many conclusions:
1) What was the policy used in exp1 for policy evaluation in MC?
2) Why Fourier basis features?
3) In MC with DQN how did you adjust the parameters and architecture for the MC task?
4) Was the reward in MC and Acrobot -1 per step or something else
5) How did you tune the parameters in the MC and Acrobot experiments?
6) Why so few runs in MC, none of the results presented are significant?
7) Why is the performance so bad in MC?
8) Did you evaluate online learning or do tests with the greedy policy?
9) How did you initialize the value functions and weights?
10) Why did you use experience replay for the linear experiments?
11) IN MC and Acrobot why only a one layer MLP?
Ignoring all that, the results are not convincing. Most of the results in the paper are not statistically significant. The policy evaluation results in MC show little difference to regular TD. The Pong results show DQN is actually better. This makes the reader wonder if the result with DQN on MC and Acrobot are only worse because you did not properly tune DQN for those domains, whereas the default DQN architecture is well tuned for Atari and that is why you method is competitive in the smaller domains. 
The differences in the “average change in value plots” are very small if the rewards are -1 per step. Can you provide some context to understand the significance of this difference? In the last experiment linear FA and MC, the step-size is set equal for all methods—this is not a valid comparison. Your method may just work better with alpha = 0.1. 
The paper has many imprecise parts, here are a few:
1) The definition of the value function would be approximate not equals unless you specify some properties of the function approximation architecture. Same for the Bellman equation
2) equation 1 of section 2.1 is neither an algorithm or a loss function
3) TD does not minimize the squared TD. Saying that is the objective function of TD learning in not true
4) end of section 2.1 says “It is computed as” but the following equation just gives a form for the partial derivative
5) equation 2, x is not bounded 
6) You state TC-loss has an unclear solution property, I don’t know what that means and I don’t think your approach is well justified either
7) Section 4.1 assumes linear FA, but its implied up until paragraph 2 that it has not assumed linear
8) treatment of n_t in alg differs from appendix (t is no time episode number)
9) Your method has a n_t parameter that is adapted according to a schedule seemingly giving it an unfair advantage over DQN.
10) Over-claim not supported by the results: “we see that HR-TD is able to find a representation that is better at keeping the target value separate than TC is “. The results do not show this.
11) Section 4.4 does not seem to go anywhere or produce and tangible conclusions
Things to improve the paper that did not impact the score:
0) It’s hard to follow how the prox operator is used in the development of the alg, this could use some higher level explaination
1) Intro p2 is about bootstrapping, use that term and remove the equations
2) Its not clear why you are talking about stochastic vs deterministic in P3
3) Perhaps you should compare against a MC method in the experiments to demonstrate the problem with TD methods and generalization
4) Section 2: “can often be a regularization term” >> can or must be?
5) update law is a odd term
6)” tends to alleviate” >> odd phrase
7) section 4 should come before section 3
8) Alg 1 in not helpful because it just references an equation
9) section 4.4 is very confusing, I cannot follow the logic of the statements 
10) Q learning >> Q-learning
11) Not sure what you mean with the last sentence of p2 section 5
12) where are the results for Acrobot linear function approximation
13) appendix Q-learning with linear FA is not DQN (table 2)