Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym*s MuJoCo continuous control environments.
The paper proposes using a first-order Taylor expansion as a control variate in policy gradient-style methods. Empirical results in dynamical control tasks suggest that this algorithm reduces the sample complexity, while the theoretical results presented suggest the algorithm is unbiased but of lower variance. The use of control variates is very important and the present paper is an interesting approach in this direction. I am not fully convinced of the approach, because it is one of many possible, and the theoretical analysis relies on an approximation of the variance rather than exact calculations, which makes it less compelling. However, this paper is a step in the right direction so it is worth accepting. In the experiments, a few things need to be discussed further: - What is the running time of the proposed approach? The computation of the extra terms required looks like it could be expensive. running time comparison in addition to sample comparison should be included - The sensitivity to parameter settings of the proposed algorithm needs to be illustrated in separate graphs, since this is one of the main claims in the paper - It would be nice to have a toy example included in which one can actually compute exact values and plot learning curves to compare more directly bias and variance. It would especially be nice to do this with a task that includes rare states, which is the case in which variance of other methods (eg importance sampling) really becomes significant.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym*s MuJoCo continuous control environments.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes using a first-order Taylor expansion as a control variate in policy gradient-style methods. Empirical results in dynamical control tasks suggest that this algorithm reduces the sample complexity, while the theoretical results presented suggest the algorithm is unbiased but of lower variance. The use of control variates is very important and the present paper is an interesting approach in this direction. I am not fully convinced of the approach, because it is one of many possible, and the theoretical analysis relies on an approximation of the variance rather than exact calculations, which makes it less compelling. However, this paper is a step in the right direction so it is worth accepting. In the experiments, a few things need to be discussed further: - What is the running time of the proposed approach? The computation of the extra terms required looks like it could be expensive. running time comparison in addition to sample comparison should be included - The sensitivity to parameter settings of the proposed algorithm needs to be illustrated in separate graphs, since this is one of the main claims in the paper - It would be nice to have a toy example included in which one can actually compute exact values and plot learning curves to compare more directly bias and variance. It would especially be nice to do this with a task that includes rare states, which is the case in which variance of other methods (eg importance sampling) really becomes significant.
assistant
The paper proposes using a first-order Taylor expansion as a control variate in policy gradient-style methods. Empirical results in dynamical control tasks suggest that this algorithm reduces the sample complexity, while the theoretical results presented suggest the algorithm is unbiased but of lower variance. The use of control variates is very important and the present paper is an interesting approach in this direction. I am not fully convinced of the approach, because it is one of many possible, and the theoretical analysis relies on an approximation of the variance rather than exact calculations, which makes it less compelling. However, this paper is a step in the right direction so it is worth accepting. In the experiments, a few things need to be discussed further: - What is the running time of the proposed approach? The computation of the extra terms required looks like it could be expensive. running time comparison in addition to sample comparison should be included - The sensitivity to parameter settings of the proposed algorithm needs to be illustrated in separate graphs, since this is one of the main claims in the paper - It would be nice to have a toy example included in which one can actually compute exact values and plot learning curves to compare more directly bias and variance. It would especially be nice to do this with a task that includes rare states, which is the case in which variance of other methods (eg importance sampling) really becomes significant.