Answers.Body: <p><em>Is it possible to use a replay buffer with TRPO?</em></p>
  <p><strong>YES</strong></p>
  <p>You can check out the ACER algorithm (<a href="https://arxiv.org/pdf/1611.01224.pdf" rel="nofollow noreferrer">paper</a> | <a href="https://github.com/openai/baselines/tree/master/baselines/acer" rel="nofollow noreferrer">code</a>)</p>
  <p>Given the formula for the policy gradient:
  <span class="math-container">$$ \nabla J(\theta) = \mathbb{E}_{a,s \sim \pi_\theta} \Big[ \nabla \log \pi_\theta (a|s) R(s,a) \Big], $$</span>
  there are two questions that need to be answered:</p>
  <ol>
  <li>How to compute the return <span class="math-container">$R(s,t)$</span> ?</li>
  <li>How to update the policy weights with this gradient ?</li>
  </ol>
  <p>TRPO is an algorithm that deals with the second question.<br />
  Instead of updating the weights in the direction of the gradient (vanilla PG),TRPO suggests updating the weights using the natural gradient. <br />
  Regarding the first question there are a few options:</p>
  <ul>
  <li>Monte-Carlo estimate: <span class="math-container">$R(s_t, a_t) = \sum_{i=t}^{T} r_{i+1}$</span>;</li>
  <li>one-step bootstrap using a value network: <span class="math-container">$R(s_t, a_t) = r_{t+1} + V_\phi (s_{t+1})$</span>;</li>
  <li>using q-value network: <span class="math-container">$R(s_t, a_t) = Q_\psi(s_t, a_t) $</span>.</li>
  </ul>
  <p>In case you decide to use a <span class="math-container">$Q$</span>-function, then you can fit that function using standard techniques from Q-learning, e.g. replay buffer, double learning and so on. You can also check this out: <a href="https://youtu.be/7C2DSdXX-kQ?t=449&amp;si=xl1nbYUMyE-F_QWU" rel="nofollow noreferrer">https://youtu.be/7C2DSdXX-kQ?t=449&amp;si=xl1nbYUMyE-F_QWU</a></p>
  <p><em>Can you train with TRPO for multiple iterations just like PPO?</em></p>
  <p><strong>NO</strong></p>
  <p>PPO and TRPO are two different algorithms that try to achieve the same goal. Given the current data that you have, what is the biggest possible update step you can take on the policy parameters. TRPO follows the natural gradient and dynamically adjust the learning rate by solving a second order equation. PPO applies multiple update iterations and clips the updates if they are too large.</p>
  <p>If you want to read more about PPO, feel free to check this blog post that I wrote:<br />
  <a href="https://pi-tau.github.io/posts/actor-critic/#ppo" rel="nofollow noreferrer">https://pi-tau.github.io/posts/actor-critic/#ppo</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-26T16:17:07.573Z
Answers.LastActivityDate: 2023-09-26T16:38:15.137Z
Answers.LastEditDate: 2023-09-26T16:38:15.137Z
Answers.UserDisplayName: pi-tau
Body: <p>I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy <span class="math-container">$\pi$</span>.</p>
  <p>Is it possible to use a replay buffer where the policy can be trained with returns gathered with old policies?</p>
  <p>Would importance sampling and the KL divergence constraint prevent the policy from making bad updates?</p>
  <p>Also, in PPO, it is stated in the algorithm that the policy is trained with the same trajectories over multiple iterations. Is this applicable for TRPO as well, seeing both algorithms are relatively similar?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-21T17:58:42.970Z
Comments.Text: It seems you're asking multiple distinct, although somehow related, questions here. I would recommend that you focus on one question in this post and remove the others, and move the others to their own separate post.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><proximal-policy-optimization><experience-replay><on-policy-methods><trust-region-policy-optimization>
Title: Can TRPO use replay buffers?
UserDisplayName: Wj210
ThreadText: # Can TRPO use replay buffers?
  
  (Asked by: username_0 on 2022-09-07)
  
  I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.
  
  Is it possible to use a replay buffer where the policy can be trained with returns gathered with old policies?
  
  Would importance sampling and the KL divergence constraint prevent the policy from making bad updates?
  
  Also, in PPO, it is stated in the algorithm that the policy is trained with the same trajectories over multiple iterations. Is this applicable for TRPO as well, seeing both algorithms are relatively similar?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, experience-replay, on-policy-methods, trust-region-policy-optimization
  > **Comments:**
  > **username_1**: It seems you're asking multiple distinct, although somehow related, questions here. I would recommend that you focus on one question in this post and remove the others, and move the others to their own separate post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-26. Score: -1
  
  *Is it possible to use a replay buffer with TRPO?*
  
  **YES**
  
  You can check out the ACER algorithm ([paper](https://arxiv.org/pdf/1611.01224.pdf) | [code](https://github.com/openai/baselines/tree/master/baselines/acer))
  
  Given the formula for the policy gradient:
  $$ \nabla J(\theta) = \mathbb{E}\_{a,s \sim \pi\_\theta} \Big[ \nabla \log \pi\_\theta (a|s) R(s,a) \Big], $$
  there are two questions that need to be answered:
  
  1. How to compute the return $R(s,t)$ ?
  2. How to update the policy weights with this gradient ?
  
  TRPO is an algorithm that deals with the second question.  
  Instead of updating the weights in the direction of the gradient (vanilla PG),TRPO suggests updating the weights using the natural gradient.   
  Regarding the first question there are a few options:
  
  * Monte-Carlo estimate: $R(s\_t, a\_t) = \sum\_{i=t}^{T} r\_{i+1}$;
  * one-step bootstrap using a value network: $R(s\_t, a\_t) = r\_{t+1} + V\_\phi (s\_{t+1})$;
  * using q-value network: $R(s\_t, a\_t) = Q\_\psi(s\_t, a\_t) $.
  
  In case you decide to use a $Q$-function, then you can fit that function using standard techniques from Q-learning, e.g. replay buffer, double learning and so on. You can also check this out: <https://youtu.be/7C2DSdXX-kQ?t=449&si=xl1nbYUMyE-F_QWU>
  
  *Can you train with TRPO for multiple iterations just like PPO?*
  
  **NO**
  
  PPO and TRPO are two different algorithms that try to achieve the same goal. Given the current data that you have, what is the biggest possible update step you can take on the policy parameters. TRPO follows the natural gradient and dynamically adjust the learning rate by solving a second order equation. PPO applies multiple update iterations and clips the updates if they are too large.
  
  If you want to read more about PPO, feel free to check this blog post that I wrote:  
  <https://pi-tau.github.io/posts/actor-critic/#ppo>
  
  
  ---
  
  