Learning to Mix n-Step Returns: Generalizing Lambda-Returns for Deep Reinforcement Learning
Reinforcement Learning (RL) can model complex behavior policies for goal-directed sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using the next state*s value function. lambda-returns define the target of the RL agent as a weighted combination of rewards estimated by using multiple many-step look-aheads. Although mathematically tractable, the use of exponentially decaying weighting of n-step returns based targets in lambda-returns is a rather ad-hoc design choice. Our major contribution is that we propose a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. In contrast to lambda-returns wherein the RL agent is restricted to use an exponentially decaying weighting scheme, CAR allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. Our experiments, in addition to showing the efficacy of CAR, also empirically demonstrate that using sophisticated weighted mixtures of multi-step returns (like CAR and lambda-returns) considerably outperforms the use of n-step returns. We perform our experiments on the Asynchronous Advantage Actor Critic (A3C) algorithm in the Atari 2600 domain.
 The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state. The key to being able to learn these confidence values is to not allow the error of the confidence estimates propagate back though the deep learning architecture. However, the method by which these confidence estimates are refined could be better described. The authors describe these confidences variously as: "some notion of confidence that the agent has in the value function estimate" and "weighing the returns based on a notion of confidence has been explored earlier (White & White, 2016; Thomas et al., 2015)". But the exact method is difficult to piece together from what is written. I believe that the confidence estimates are considered to be part of the critic and the w vector to be part of the theta_c parameters. This would then be captured by the critic gradient for the CAR method that appears towards the end of page 5. If so, this should be stated explicitly. There is another theoretical point that could be clearer. The variation in an autodidactic update of a value function (Equation (4)) depends on a few things, the in variation future value function estimates themselves being just one factor. Another two sources of variation are: the uncertainty over how likely each path is to be taken, and the uncertainty in immediate rewards accumulated as part of some n-step return. In my opinion, the quality of the paper would be much improved by a brief discussion of this, and some reflection on what aspects of these variation contribute to the confidence vectors and what isn*t captured. Nonetheless, I believe that the paper represents an interesting and worthy submission to the conference. I would strongly urge the authors to improve the method description in the camera read version though. A few additional comments are as follows: • The plot in Figure 3 is the leading collection of results to demonstrate the dominance of the authors* adaptive weight approach (CAR) over the A3C (TD(0) estimates) and LRA3C (truncated TD(lambda) estimates) approaches. However, the way the results are presented/plotted, namely the linear plot of the (shifted) relative performance of CAR (and LRA3C) versus A3C, visually inflates the importance of tasks on which CAR (and LRA3C) perform better than A3C, and diminishes the importance of those tasks on which A3C performs better. It would be better kept as a relative value and plotted on a log-scale so that positive and negative improvements can be viewed on an equal setting. • On page 3, when Gt is first mentioned, Gt should really be described first, before the reader is told what it is often replaced with. • On page 3, where delta_t is defined (the j step return TD error, I think the middle term should be • On page 4 and 5, when describing the gradient for the actor and critic, it would be better if these were given their own terminology, but if not, then use of the word respectively in each case would help.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning to Mix n-Step Returns: Generalizing Lambda-Returns for Deep Reinforcement Learning
Reinforcement Learning (RL) can model complex behavior policies for goal-directed sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using the next state*s value function. lambda-returns define the target of the RL agent as a weighted combination of rewards estimated by using multiple many-step look-aheads. Although mathematically tractable, the use of exponentially decaying weighting of n-step returns based targets in lambda-returns is a rather ad-hoc design choice. Our major contribution is that we propose a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. In contrast to lambda-returns wherein the RL agent is restricted to use an exponentially decaying weighting scheme, CAR allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. Our experiments, in addition to showing the efficacy of CAR, also empirically demonstrate that using sophisticated weighted mixtures of multi-step returns (like CAR and lambda-returns) considerably outperforms the use of n-step returns. We perform our experiments on the Asynchronous Advantage Actor Critic (A3C) algorithm in the Atari 2600 domain.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state. The key to being able to learn these confidence values is to not allow the error of the confidence estimates propagate back though the deep learning architecture. However, the method by which these confidence estimates are refined could be better described. The authors describe these confidences variously as: "some notion of confidence that the agent has in the value function estimate" and "weighing the returns based on a notion of confidence has been explored earlier (White & White, 2016; Thomas et al., 2015)". But the exact method is difficult to piece together from what is written. I believe that the confidence estimates are considered to be part of the critic and the w vector to be part of the theta_c parameters. This would then be captured by the critic gradient for the CAR method that appears towards the end of page 5. If so, this should be stated explicitly. There is another theoretical point that could be clearer. The variation in an autodidactic update of a value function (Equation (4)) depends on a few things, the in variation future value function estimates themselves being just one factor. Another two sources of variation are: the uncertainty over how likely each path is to be taken, and the uncertainty in immediate rewards accumulated as part of some n-step return. In my opinion, the quality of the paper would be much improved by a brief discussion of this, and some reflection on what aspects of these variation contribute to the confidence vectors and what isn*t captured. Nonetheless, I believe that the paper represents an interesting and worthy submission to the conference. I would strongly urge the authors to improve the method description in the camera read version though. A few additional comments are as follows: • The plot in Figure 3 is the leading collection of results to demonstrate the dominance of the authors* adaptive weight approach (CAR) over the A3C (TD(0) estimates) and LRA3C (truncated TD(lambda) estimates) approaches. However, the way the results are presented/plotted, namely the linear plot of the (shifted) relative performance of CAR (and LRA3C) versus A3C, visually inflates the importance of tasks on which CAR (and LRA3C) perform better than A3C, and diminishes the importance of those tasks on which A3C performs better. It would be better kept as a relative value and plotted on a log-scale so that positive and negative improvements can be viewed on an equal setting. • On page 3, when Gt is first mentioned, Gt should really be described first, before the reader is told what it is often replaced with. • On page 3, where delta_t is defined (the j step return TD error, I think the middle term should be • On page 4 and 5, when describing the gradient for the actor and critic, it would be better if these were given their own terminology, but if not, then use of the word respectively in each case would help.
assistant
 The authors present confidence-based autodidactic returns, a Deep learning RL method to adjust the weights of an eligibility vector in TD(lambda)-like value estimation to favour more stable estimates of the state. The key to being able to learn these confidence values is to not allow the error of the confidence estimates propagate back though the deep learning architecture. However, the method by which these confidence estimates are refined could be better described. The authors describe these confidences variously as: "some notion of confidence that the agent has in the value function estimate" and "weighing the returns based on a notion of confidence has been explored earlier (White & White, 2016; Thomas et al., 2015)". But the exact method is difficult to piece together from what is written. I believe that the confidence estimates are considered to be part of the critic and the w vector to be part of the theta_c parameters. This would then be captured by the critic gradient for the CAR method that appears towards the end of page 5. If so, this should be stated explicitly. There is another theoretical point that could be clearer. The variation in an autodidactic update of a value function (Equation (4)) depends on a few things, the in variation future value function estimates themselves being just one factor. Another two sources of variation are: the uncertainty over how likely each path is to be taken, and the uncertainty in immediate rewards accumulated as part of some n-step return. In my opinion, the quality of the paper would be much improved by a brief discussion of this, and some reflection on what aspects of these variation contribute to the confidence vectors and what isn*t captured. Nonetheless, I believe that the paper represents an interesting and worthy submission to the conference. I would strongly urge the authors to improve the method description in the camera read version though. A few additional comments are as follows: • The plot in Figure 3 is the leading collection of results to demonstrate the dominance of the authors* adaptive weight approach (CAR) over the A3C (TD(0) estimates) and LRA3C (truncated TD(lambda) estimates) approaches. However, the way the results are presented/plotted, namely the linear plot of the (shifted) relative performance of CAR (and LRA3C) versus A3C, visually inflates the importance of tasks on which CAR (and LRA3C) perform better than A3C, and diminishes the importance of those tasks on which A3C performs better. It would be better kept as a relative value and plotted on a log-scale so that positive and negative improvements can be viewed on an equal setting. • On page 3, when Gt is first mentioned, Gt should really be described first, before the reader is told what it is often replaced with. • On page 3, where delta_t is defined (the j step return TD error, I think the middle term should be • On page 4 and 5, when describing the gradient for the actor and critic, it would be better if these were given their own terminology, but if not, then use of the word respectively in each case would help.