Two-Timescale Networks for Nonlinear Value Function Approximation
A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approximation---with a fixed basis or fixed representation. Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning. In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale. The approach facilitates the use of algorithms developed for the linear setting, such as data-efficient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates. We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation. We empirically demonstrate the benefits of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.
 This paper proposes Two-Timescale Networks (TTNs), a reinforcement learning algorithm where feature representations are learned by a neural network trained on a surrogate loss function (i.e. value), and a value function is learned on top of the feature representation using a "fast" least-squares algorithm. The authors prove the convergence of this method using methods from two time-scale stochastic approximation. Convergent and stable nonlinear algorithms is an important problem in reinforcement learning, and this paper offers an interesting approach for addressing this issue. The idea of using a "fast" linear learner on top of a slowly changing representation is not new in RL (Levine et. al, 2017), but the authors somewhat motivate this approach by showing that it results in a stable and convergent algorithm. Thus, I view the convergence proof as the main contribution of the paper. The paper is written clearly, but could benefit from more efficient use of space in the main paper. For example, I feel that the introduction and discussion in Section 3 on surrogate objectives could be considerably shortened, and a formal proof statement could be included from the appendix in Section 4, with the full proof in the appendix. The experimental evaluation is detailed, and ablation tests show the value of different choices of surrogate loss for value function training, linear value function learning methods, and comparisons against other nonlinear algorithms such as DQN and Nonlinear GTD/TD/variants. A minor criticism is that it is difficult to position this work against the "simpler but not sound" deep RL methods, as the authors only compare to DQN on a non-standard benchmark task. As additional related work, SBEED (Dai et. al, ICML 2018) also shows convergence for a nonlinear reinforcement learning algorithm (in the control setting), and quantifies the convergence rate while accounting for finite sample error. It would be good to include discussion of this work, although the proposed method and proofs are derived very differently.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Two-Timescale Networks for Nonlinear Value Function Approximation
A key component for many reinforcement learning agents is to learn a value function, either for policy evaluation or control. Many of the algorithms for learning values, however, are designed for linear function approximation---with a fixed basis or fixed representation. Though there have been a few sound extensions to nonlinear function approximation, such as nonlinear gradient temporal difference learning, these methods have largely not been adopted, eschewed in favour of simpler but not sound methods like temporal difference learning and Q-learning. In this work, we provide a two-timescale network (TTN) architecture that enables linear methods to be used to learn values, with a nonlinear representation learned at a slower timescale. The approach facilitates the use of algorithms developed for the linear setting, such as data-efficient least-squares methods, eligibility traces and the myriad of recently developed linear policy evaluation algorithms, to provide nonlinear value estimates. We prove convergence for TTNs, with particular care given to ensure convergence of the fast linear component under potentially dependent features provided by the learned representation. We empirically demonstrate the benefits of TTNs, compared to other nonlinear value function approximation algorithms, both for policy evaluation and control.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes Two-Timescale Networks (TTNs), a reinforcement learning algorithm where feature representations are learned by a neural network trained on a surrogate loss function (i.e. value), and a value function is learned on top of the feature representation using a "fast" least-squares algorithm. The authors prove the convergence of this method using methods from two time-scale stochastic approximation. Convergent and stable nonlinear algorithms is an important problem in reinforcement learning, and this paper offers an interesting approach for addressing this issue. The idea of using a "fast" linear learner on top of a slowly changing representation is not new in RL (Levine et. al, 2017), but the authors somewhat motivate this approach by showing that it results in a stable and convergent algorithm. Thus, I view the convergence proof as the main contribution of the paper. The paper is written clearly, but could benefit from more efficient use of space in the main paper. For example, I feel that the introduction and discussion in Section 3 on surrogate objectives could be considerably shortened, and a formal proof statement could be included from the appendix in Section 4, with the full proof in the appendix. The experimental evaluation is detailed, and ablation tests show the value of different choices of surrogate loss for value function training, linear value function learning methods, and comparisons against other nonlinear algorithms such as DQN and Nonlinear GTD/TD/variants. A minor criticism is that it is difficult to position this work against the "simpler but not sound" deep RL methods, as the authors only compare to DQN on a non-standard benchmark task. As additional related work, SBEED (Dai et. al, ICML 2018) also shows convergence for a nonlinear reinforcement learning algorithm (in the control setting), and quantifies the convergence rate while accounting for finite sample error. It would be good to include discussion of this work, although the proposed method and proofs are derived very differently.
assistant
 This paper proposes Two-Timescale Networks (TTNs), a reinforcement learning algorithm where feature representations are learned by a neural network trained on a surrogate loss function (i.e. value), and a value function is learned on top of the feature representation using a "fast" least-squares algorithm. The authors prove the convergence of this method using methods from two time-scale stochastic approximation. Convergent and stable nonlinear algorithms is an important problem in reinforcement learning, and this paper offers an interesting approach for addressing this issue. The idea of using a "fast" linear learner on top of a slowly changing representation is not new in RL (Levine et. al, 2017), but the authors somewhat motivate this approach by showing that it results in a stable and convergent algorithm. Thus, I view the convergence proof as the main contribution of the paper. The paper is written clearly, but could benefit from more efficient use of space in the main paper. For example, I feel that the introduction and discussion in Section 3 on surrogate objectives could be considerably shortened, and a formal proof statement could be included from the appendix in Section 4, with the full proof in the appendix. The experimental evaluation is detailed, and ablation tests show the value of different choices of surrogate loss for value function training, linear value function learning methods, and comparisons against other nonlinear algorithms such as DQN and Nonlinear GTD/TD/variants. A minor criticism is that it is difficult to position this work against the "simpler but not sound" deep RL methods, as the authors only compare to DQN on a non-standard benchmark task. As additional related work, SBEED (Dai et. al, ICML 2018) also shows convergence for a nonlinear reinforcement learning algorithm (in the control setting), and quantifies the convergence rate while accounting for finite sample error. It would be good to include discussion of this work, although the proposed method and proofs are derived very differently.