Samples Are Useful? Not Always: denoising policy gradient updates using variance explained
Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based solely on sampling the agent*s policy, other measures directly accessible through these algorithms could be used to improve sampling before each policy update. Following this line of thoughts, we propose the use of SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. This criterion, the fraction of variance explained Vex, is a measure of the discrepancy between a model and actual samples. In this work, Vex is used to evaluate the impact each transition will have on learning: this criterion refines sampling and improves the policy gradient algorithm. In this paper: (a) We introduce and explore Vex, the criterion used for denoising policy gradient updates. (b) We conduct experiments across a variety of benchmark environments, including standard continuous control problems. Our results show better performance with SAUNA. (c) We investigate why Vex provides a reliable assessment for the selection of samples that will positively impact learning. (d) We show how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.
In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach. The revision is a significantly better paper than the original, and I am updating my score accordingly. ------------- This paper introduces *the fraction of variance explained* V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function. In this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory. Intuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors* assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I*m also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below). The paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper. With the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I*m not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure. Questions for the authors: - what proportion of samples are rejected - how does this proportion change over the course of learning - how was the V^{ex} threshold of 0.3 chosen
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Samples Are Useful? Not Always: denoising policy gradient updates using variance explained
Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based solely on sampling the agent*s policy, other measures directly accessible through these algorithms could be used to improve sampling before each policy update. Following this line of thoughts, we propose the use of SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. This criterion, the fraction of variance explained Vex, is a measure of the discrepancy between a model and actual samples. In this work, Vex is used to evaluate the impact each transition will have on learning: this criterion refines sampling and improves the policy gradient algorithm. In this paper: (a) We introduce and explore Vex, the criterion used for denoising policy gradient updates. (b) We conduct experiments across a variety of benchmark environments, including standard continuous control problems. Our results show better performance with SAUNA. (c) We investigate why Vex provides a reliable assessment for the selection of samples that will positively impact learning. (d) We show how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach. The revision is a significantly better paper than the original, and I am updating my score accordingly. ------------- This paper introduces *the fraction of variance explained* V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function. In this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory. Intuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors* assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I*m also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below). The paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper. With the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I*m not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure. Questions for the authors: - what proportion of samples are rejected - how does this proportion change over the course of learning - how was the V^{ex} threshold of 0.3 chosen
assistant
In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach. The revision is a significantly better paper than the original, and I am updating my score accordingly. ------------- This paper introduces *the fraction of variance explained* V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function. In this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory. Intuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors* assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I*m also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below). The paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper. With the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I*m not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure. Questions for the authors: - what proportion of samples are rejected - how does this proportion change over the course of learning - how was the V^{ex} threshold of 0.3 chosen