TD Learning with Constrained Gradients
Temporal Difference Learning with function approximation is known to be unstable. Previous work like citet{sutton2009fast} and citet{sutton2009convergent} has presented alternative objectives that are stable to minimize. However, in practice, TD-learning with neural networks requires various tricks like using a target network that updates slowly citep{mnih2015human}. In this work we propose a constraint on the TD update that minimizes change to the target values. This constraint can be applied to the gradients of any TD objective, and can be easily applied to nonlinear function approximation. We validate this update by applying our technique to deep Q-learning, and training without a target network. We also show that adding this constraint on Baird*s counterexample keeps Q-learning from diverging.
 This is an interesting idea, and written clearly. The experiments with Baird*s and CartPole were both convincing as preliminary evidence that this could be effective. However, it is very hard to generalize from these toy problems. First, we really need a more thorough analysis of what this does to the learning dynamics itself. Baring theoretical results, you could analyze the changes to the value function at the current and next state with and without the constraint to illustrate the effects more directly. I think ideally, I would want to see this on Atari or some of the continuous control domains often used. If this allows the removing of the target network for instance, in those more difficult tasks, then this would be a huge deal. Additionally, I do not think the current gridworld task adds anything to the experiments, I would rather actually see this on a more interesting linear function approximation on some other simple task like Mountain Car than a neural network on gridworld. The reason this might be interesting is that when the parameter space is lower dimensional (not an issue for neural nets, but could be problematic for linear FA) the constraint might be too much leading to significantly poorer performance. I suspect this is the actual cause for it not converging to zero for Baird*s, although please correct me if I*m wrong on that. As is, I cannot recommend acceptance given the current experiments and lack of theoretical results. But I do think this is a very interesting direction and hope to see more thorough experiments or analysis to support it. Pros: Simple, interesting idea Works well on toy problems, and able to prevent divergence in Baird*s counter-example Cons: Lacking in theoretical analysis or significant experimental results
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
TD Learning with Constrained Gradients
Temporal Difference Learning with function approximation is known to be unstable. Previous work like citet{sutton2009fast} and citet{sutton2009convergent} has presented alternative objectives that are stable to minimize. However, in practice, TD-learning with neural networks requires various tricks like using a target network that updates slowly citep{mnih2015human}. In this work we propose a constraint on the TD update that minimizes change to the target values. This constraint can be applied to the gradients of any TD objective, and can be easily applied to nonlinear function approximation. We validate this update by applying our technique to deep Q-learning, and training without a target network. We also show that adding this constraint on Baird*s counterexample keeps Q-learning from diverging.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This is an interesting idea, and written clearly. The experiments with Baird*s and CartPole were both convincing as preliminary evidence that this could be effective. However, it is very hard to generalize from these toy problems. First, we really need a more thorough analysis of what this does to the learning dynamics itself. Baring theoretical results, you could analyze the changes to the value function at the current and next state with and without the constraint to illustrate the effects more directly. I think ideally, I would want to see this on Atari or some of the continuous control domains often used. If this allows the removing of the target network for instance, in those more difficult tasks, then this would be a huge deal. Additionally, I do not think the current gridworld task adds anything to the experiments, I would rather actually see this on a more interesting linear function approximation on some other simple task like Mountain Car than a neural network on gridworld. The reason this might be interesting is that when the parameter space is lower dimensional (not an issue for neural nets, but could be problematic for linear FA) the constraint might be too much leading to significantly poorer performance. I suspect this is the actual cause for it not converging to zero for Baird*s, although please correct me if I*m wrong on that. As is, I cannot recommend acceptance given the current experiments and lack of theoretical results. But I do think this is a very interesting direction and hope to see more thorough experiments or analysis to support it. Pros: Simple, interesting idea Works well on toy problems, and able to prevent divergence in Baird*s counter-example Cons: Lacking in theoretical analysis or significant experimental results
assistant
 This is an interesting idea, and written clearly. The experiments with Baird*s and CartPole were both convincing as preliminary evidence that this could be effective. However, it is very hard to generalize from these toy problems. First, we really need a more thorough analysis of what this does to the learning dynamics itself. Baring theoretical results, you could analyze the changes to the value function at the current and next state with and without the constraint to illustrate the effects more directly. I think ideally, I would want to see this on Atari or some of the continuous control domains often used. If this allows the removing of the target network for instance, in those more difficult tasks, then this would be a huge deal. Additionally, I do not think the current gridworld task adds anything to the experiments, I would rather actually see this on a more interesting linear function approximation on some other simple task like Mountain Car than a neural network on gridworld. The reason this might be interesting is that when the parameter space is lower dimensional (not an issue for neural nets, but could be problematic for linear FA) the constraint might be too much leading to significantly poorer performance. I suspect this is the actual cause for it not converging to zero for Baird*s, although please correct me if I*m wrong on that. As is, I cannot recommend acceptance given the current experiments and lack of theoretical results. But I do think this is a very interesting direction and hope to see more thorough experiments or analysis to support it. Pros: Simple, interesting idea Works well on toy problems, and able to prevent divergence in Baird*s counter-example Cons: Lacking in theoretical analysis or significant experimental results