TD Learning with Constrained Gradients
Temporal Difference Learning with function approximation is known to be unstable. Previous work like citet{sutton2009fast} and citet{sutton2009convergent} has presented alternative objectives that are stable to minimize. However, in practice, TD-learning with neural networks requires various tricks like using a target network that updates slowly citep{mnih2015human}. In this work we propose a constraint on the TD update that minimizes change to the target values. This constraint can be applied to the gradients of any TD objective, and can be easily applied to nonlinear function approximation. We validate this update by applying our technique to deep Q-learning, and training without a target network. We also show that adding this constraint on Baird*s counterexample keeps Q-learning from diverging.
 Summary: This paper tackles the issue of combining TD learning methods with function approximation. The proposed algorithm constrains the gradient update to deal with the fact that canonical TD with function approximation ignores the impact of changing the weights on the target of the TD learning rule. Results with linear and non-linear function approximation highlight the attributes of the method. Quality: The quality of the writing, notation, motivation, and results analysis is low. I will give a few examples to highlight the point. The paper motivates that TD is divergent with function approximation, and then goes on to discuss MSPBE methods that have strong convergence results, without addressing why a new approach is needed. There are many missing references: ETD, HTD, mirror-prox methods, retrace, ABQ. Q-sigma. This is a very active area of research and the paper needs to justify their approach. The paper has straightforward technical errors and naive statements: e.g. the equation for the loss of TD takes the norm of a scalar. The paper claims that it is not well-known that TD with function approximation ignores part of the gradient of the MSVE. There are many others. The experiments have serious issues. Exp1 seems to indicate that the new method does not converge to the correct solution. The grid world experiment is not conclusive as important details like the number of episodes and how parameters were chosen was not discussed. Again exp3 provides little information about the experimental setup. Clarity: The clarity of the text is fine, though errors make things difficult sometimes. For example The Bhatnagar 2009 reference should be Maei. Originality: As mentioned above this is a very active research area, and the paper makes little effort to explain why the multitude of existing algorithms are not suitable. Significance: Because of all the things outlined above, the significance is below the bar for this round.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
TD Learning with Constrained Gradients
Temporal Difference Learning with function approximation is known to be unstable. Previous work like citet{sutton2009fast} and citet{sutton2009convergent} has presented alternative objectives that are stable to minimize. However, in practice, TD-learning with neural networks requires various tricks like using a target network that updates slowly citep{mnih2015human}. In this work we propose a constraint on the TD update that minimizes change to the target values. This constraint can be applied to the gradients of any TD objective, and can be easily applied to nonlinear function approximation. We validate this update by applying our technique to deep Q-learning, and training without a target network. We also show that adding this constraint on Baird*s counterexample keeps Q-learning from diverging.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary: This paper tackles the issue of combining TD learning methods with function approximation. The proposed algorithm constrains the gradient update to deal with the fact that canonical TD with function approximation ignores the impact of changing the weights on the target of the TD learning rule. Results with linear and non-linear function approximation highlight the attributes of the method. Quality: The quality of the writing, notation, motivation, and results analysis is low. I will give a few examples to highlight the point. The paper motivates that TD is divergent with function approximation, and then goes on to discuss MSPBE methods that have strong convergence results, without addressing why a new approach is needed. There are many missing references: ETD, HTD, mirror-prox methods, retrace, ABQ. Q-sigma. This is a very active area of research and the paper needs to justify their approach. The paper has straightforward technical errors and naive statements: e.g. the equation for the loss of TD takes the norm of a scalar. The paper claims that it is not well-known that TD with function approximation ignores part of the gradient of the MSVE. There are many others. The experiments have serious issues. Exp1 seems to indicate that the new method does not converge to the correct solution. The grid world experiment is not conclusive as important details like the number of episodes and how parameters were chosen was not discussed. Again exp3 provides little information about the experimental setup. Clarity: The clarity of the text is fine, though errors make things difficult sometimes. For example The Bhatnagar 2009 reference should be Maei. Originality: As mentioned above this is a very active research area, and the paper makes little effort to explain why the multitude of existing algorithms are not suitable. Significance: Because of all the things outlined above, the significance is below the bar for this round.
assistant
 Summary: This paper tackles the issue of combining TD learning methods with function approximation. The proposed algorithm constrains the gradient update to deal with the fact that canonical TD with function approximation ignores the impact of changing the weights on the target of the TD learning rule. Results with linear and non-linear function approximation highlight the attributes of the method. Quality: The quality of the writing, notation, motivation, and results analysis is low. I will give a few examples to highlight the point. The paper motivates that TD is divergent with function approximation, and then goes on to discuss MSPBE methods that have strong convergence results, without addressing why a new approach is needed. There are many missing references: ETD, HTD, mirror-prox methods, retrace, ABQ. Q-sigma. This is a very active area of research and the paper needs to justify their approach. The paper has straightforward technical errors and naive statements: e.g. the equation for the loss of TD takes the norm of a scalar. The paper claims that it is not well-known that TD with function approximation ignores part of the gradient of the MSVE. There are many others. The experiments have serious issues. Exp1 seems to indicate that the new method does not converge to the correct solution. The grid world experiment is not conclusive as important details like the number of episodes and how parameters were chosen was not discussed. Again exp3 provides little information about the experimental setup. Clarity: The clarity of the text is fine, though errors make things difficult sometimes. For example The Bhatnagar 2009 reference should be Maei. Originality: As mentioned above this is a very active research area, and the paper makes little effort to explain why the multitude of existing algorithms are not suitable. Significance: Because of all the things outlined above, the significance is below the bar for this round.