Off-policy Multi-step Q-learning
In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency which is limiting with regard to real-world applications. We follow the idea of multi-step TD-learning to enhance data-efficiency while remaining off-policy by proposing two novel Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a policy rollout and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout. We prove that the combination of these short- and long-term predictions is a representation of the full return, leading to the Composite Q-learning algorithm. We show the efficacy of Composite Q-learning in the tabular case and compare our approach in the function-approximation setting with TD3, Model-based Value Expansion and TD3(Delta), which we introduce as an off-policy variant of TD(Delta). We show on three simulated robot tasks that Composite TD3 outperforms TD3 as well as state-of-the-art off-policy multi-step approaches in terms of data-efficiency.
Summary This paper introduces a new Q-learning formalism that helps reduce the bias of single step bootstrapping in Q-learning by learning multiple single step bootstrapping Q functions in parallel. This is accomplished by composing multiple n-step returns, showing that a recursive definition of n-step returns allows each return to be learned using only a single step of bootstrapping instead of at most n steps of bootstrapping. The paper solves the problem of the n-step fixed horizon by additionally composing a gamma discounted Q function that is shifted by n. In the end, the Q function used for behavior still predicts the same values as vanilla Q-learning, but with significantly less bias without a large increase in variance. Review I find this paper to be novel and insightful, the proposed algorithm is well supported theoretically and reasonably well supported empirically. I appreciated the careful demonstration on the smaller MDP with tabular features, showing the effects of multi-step Q-learning and clearly demonstrating the bias due to not truly using an off-policy formulation. I find that the demonstrations on the larger environments appear promising and suggest that composite multi-step Q-learning is a promising direction. The error bars in the larger demonstrations, Figure 4, make it difficult to distinguish any meaningful differences between the algorithms. I appreciate that results are averaged over 11 runs, fortunately far more than seems to be standard at the moment, but still the amount of variance makes it difficult to say anything statistically. Table 2, then shows a reduction of the results but without mention of variance. It would be useful to include error measurements (perhaps the standard error over runs) to Table 2 to see the statistical significance of those results. Based on Figure 4, my guess is that there is negligible difference statistically. The parameter sensitivity curves for the Walker2d domain also demonstrate that it is difficult to say anything meaningful about each parameter choice. Running a larger number of parameter settings would help to establish a clear pattern, or running each parameter setting for more independent runs could have allowed more significant results. The variance exhibited by one value in the regularization sensitivity curve is alone extremely interesting; perhaps using a different visualization that allowed more clear comparisons of the variance over independent runs would further motivate the utility of the regularization parameter. I think these results are interesting, but as presented do not sufficiently highlight the differences between the proposed algorithm and its competitors. For the experiment in Figure 2, why not include multi-step Q-learning with importance sampling corrections on the later steps? I believe this would have fixed the bias issue, though clearly would be a tradeoff for high variance. I think this would make for a more convincing argument. Additionally, the caption does not well explain what the four green lines at the top of the plot represent. It was difficult to interpret the plot on the first pass of the paper because of this omission. Regardless, I find the results in Figure 2 to be otherwise intriguing. Finally, the choice of meta-parameters in this paper could negatively impact results in favor of the competitor algorithms. By choosing to fix meta-parameters based on the defaults of a competitor, this could be harmfully biasing the proposed algorithm by preventing it from choosing a better stepsize. In fact, I would suspect that the proposed algorithm would exhibit lower variance updates than TD3, meaning it could potentially take advantage of higher stepsizes. This omission makes the claims of this paper weaker than they could possibly be, leaving a slight hole in the research. --------- Edit after discussion and rebuttal phase: I read the in-depth discussion between the authors and R3 and looked at the edits to the draft. I agree with the other reviewers on the basis of understanding the importance of meta-parameter selection. During the initial review, I found the ideas of the paper interesting enough to largely out-weigh the importance of a careful meta-parameter study. After R3*s demonstration that there were indeed flaws with the results under the current meta-parameter selections, I think the best course of action would be to reject the paper in its current form. I still strongly believe there is a place in the literature for this paper, so I hope to see this paper again at the next conference.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Off-policy Multi-step Q-learning
In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency which is limiting with regard to real-world applications. We follow the idea of multi-step TD-learning to enhance data-efficiency while remaining off-policy by proposing two novel Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a policy rollout and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout. We prove that the combination of these short- and long-term predictions is a representation of the full return, leading to the Composite Q-learning algorithm. We show the efficacy of Composite Q-learning in the tabular case and compare our approach in the function-approximation setting with TD3, Model-based Value Expansion and TD3(Delta), which we introduce as an off-policy variant of TD(Delta). We show on three simulated robot tasks that Composite TD3 outperforms TD3 as well as state-of-the-art off-policy multi-step approaches in terms of data-efficiency.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary This paper introduces a new Q-learning formalism that helps reduce the bias of single step bootstrapping in Q-learning by learning multiple single step bootstrapping Q functions in parallel. This is accomplished by composing multiple n-step returns, showing that a recursive definition of n-step returns allows each return to be learned using only a single step of bootstrapping instead of at most n steps of bootstrapping. The paper solves the problem of the n-step fixed horizon by additionally composing a gamma discounted Q function that is shifted by n. In the end, the Q function used for behavior still predicts the same values as vanilla Q-learning, but with significantly less bias without a large increase in variance. Review I find this paper to be novel and insightful, the proposed algorithm is well supported theoretically and reasonably well supported empirically. I appreciated the careful demonstration on the smaller MDP with tabular features, showing the effects of multi-step Q-learning and clearly demonstrating the bias due to not truly using an off-policy formulation. I find that the demonstrations on the larger environments appear promising and suggest that composite multi-step Q-learning is a promising direction. The error bars in the larger demonstrations, Figure 4, make it difficult to distinguish any meaningful differences between the algorithms. I appreciate that results are averaged over 11 runs, fortunately far more than seems to be standard at the moment, but still the amount of variance makes it difficult to say anything statistically. Table 2, then shows a reduction of the results but without mention of variance. It would be useful to include error measurements (perhaps the standard error over runs) to Table 2 to see the statistical significance of those results. Based on Figure 4, my guess is that there is negligible difference statistically. The parameter sensitivity curves for the Walker2d domain also demonstrate that it is difficult to say anything meaningful about each parameter choice. Running a larger number of parameter settings would help to establish a clear pattern, or running each parameter setting for more independent runs could have allowed more significant results. The variance exhibited by one value in the regularization sensitivity curve is alone extremely interesting; perhaps using a different visualization that allowed more clear comparisons of the variance over independent runs would further motivate the utility of the regularization parameter. I think these results are interesting, but as presented do not sufficiently highlight the differences between the proposed algorithm and its competitors. For the experiment in Figure 2, why not include multi-step Q-learning with importance sampling corrections on the later steps? I believe this would have fixed the bias issue, though clearly would be a tradeoff for high variance. I think this would make for a more convincing argument. Additionally, the caption does not well explain what the four green lines at the top of the plot represent. It was difficult to interpret the plot on the first pass of the paper because of this omission. Regardless, I find the results in Figure 2 to be otherwise intriguing. Finally, the choice of meta-parameters in this paper could negatively impact results in favor of the competitor algorithms. By choosing to fix meta-parameters based on the defaults of a competitor, this could be harmfully biasing the proposed algorithm by preventing it from choosing a better stepsize. In fact, I would suspect that the proposed algorithm would exhibit lower variance updates than TD3, meaning it could potentially take advantage of higher stepsizes. This omission makes the claims of this paper weaker than they could possibly be, leaving a slight hole in the research. --------- Edit after discussion and rebuttal phase: I read the in-depth discussion between the authors and R3 and looked at the edits to the draft. I agree with the other reviewers on the basis of understanding the importance of meta-parameter selection. During the initial review, I found the ideas of the paper interesting enough to largely out-weigh the importance of a careful meta-parameter study. After R3*s demonstration that there were indeed flaws with the results under the current meta-parameter selections, I think the best course of action would be to reject the paper in its current form. I still strongly believe there is a place in the literature for this paper, so I hope to see this paper again at the next conference.
assistant
Summary This paper introduces a new Q-learning formalism that helps reduce the bias of single step bootstrapping in Q-learning by learning multiple single step bootstrapping Q functions in parallel. This is accomplished by composing multiple n-step returns, showing that a recursive definition of n-step returns allows each return to be learned using only a single step of bootstrapping instead of at most n steps of bootstrapping. The paper solves the problem of the n-step fixed horizon by additionally composing a gamma discounted Q function that is shifted by n. In the end, the Q function used for behavior still predicts the same values as vanilla Q-learning, but with significantly less bias without a large increase in variance. Review I find this paper to be novel and insightful, the proposed algorithm is well supported theoretically and reasonably well supported empirically. I appreciated the careful demonstration on the smaller MDP with tabular features, showing the effects of multi-step Q-learning and clearly demonstrating the bias due to not truly using an off-policy formulation. I find that the demonstrations on the larger environments appear promising and suggest that composite multi-step Q-learning is a promising direction. The error bars in the larger demonstrations, Figure 4, make it difficult to distinguish any meaningful differences between the algorithms. I appreciate that results are averaged over 11 runs, fortunately far more than seems to be standard at the moment, but still the amount of variance makes it difficult to say anything statistically. Table 2, then shows a reduction of the results but without mention of variance. It would be useful to include error measurements (perhaps the standard error over runs) to Table 2 to see the statistical significance of those results. Based on Figure 4, my guess is that there is negligible difference statistically. The parameter sensitivity curves for the Walker2d domain also demonstrate that it is difficult to say anything meaningful about each parameter choice. Running a larger number of parameter settings would help to establish a clear pattern, or running each parameter setting for more independent runs could have allowed more significant results. The variance exhibited by one value in the regularization sensitivity curve is alone extremely interesting; perhaps using a different visualization that allowed more clear comparisons of the variance over independent runs would further motivate the utility of the regularization parameter. I think these results are interesting, but as presented do not sufficiently highlight the differences between the proposed algorithm and its competitors. For the experiment in Figure 2, why not include multi-step Q-learning with importance sampling corrections on the later steps? I believe this would have fixed the bias issue, though clearly would be a tradeoff for high variance. I think this would make for a more convincing argument. Additionally, the caption does not well explain what the four green lines at the top of the plot represent. It was difficult to interpret the plot on the first pass of the paper because of this omission. Regardless, I find the results in Figure 2 to be otherwise intriguing. Finally, the choice of meta-parameters in this paper could negatively impact results in favor of the competitor algorithms. By choosing to fix meta-parameters based on the defaults of a competitor, this could be harmfully biasing the proposed algorithm by preventing it from choosing a better stepsize. In fact, I would suspect that the proposed algorithm would exhibit lower variance updates than TD3, meaning it could potentially take advantage of higher stepsizes. This omission makes the claims of this paper weaker than they could possibly be, leaving a slight hole in the research. --------- Edit after discussion and rebuttal phase: I read the in-depth discussion between the authors and R3 and looked at the edits to the draft. I agree with the other reviewers on the basis of understanding the importance of meta-parameter selection. During the initial review, I found the ideas of the paper interesting enough to largely out-weigh the importance of a careful meta-parameter study. After R3*s demonstration that there were indeed flaws with the results under the current meta-parameter selections, I think the best course of action would be to reject the paper in its current form. I still strongly believe there is a place in the literature for this paper, so I hope to see this paper again at the next conference.