Answers.Body: <p>Seems this question was asked and answered in <a href="https://github.com/openai/baselines/issues/316" rel="nofollow noreferrer">openai/baselines github issue</a>. The issue has been closed for a while.</p>
  <p>Below is an answer provided by @matthiasplappert which has the most &quot;thumbs up&quot;:</p>
  <blockquote>
  <p>To clarify: PPO is an on-policy algorithm so you are correct that going over the same data multiple times is technically incorrect.</p>
  <p>However, we found that PPO is actually quite okay with doing this and we still get stable convergence. This is likely due to the proximal trust region constrained that we enforce, which means that the policy cannot change that much anyway when going over the current set of transitions multiple times, making it still approximately on-policy. You can of course get rid of this but then you'll need more samples.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T06:15:05.313Z
Answers.LastActivityDate: 2020-12-14T06:15:05.313Z
Answers.UserDisplayName: Dawid Laszuk
Body: <p>I have a general question about the updating of the network/model in the <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">PPO algorithm</a>.</p>
  <p>If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).</p>
  <p>Basically, in the pseudo-code of the algorithm, I don't understand the line &quot;Optimize surrogate L ... with K epochs...&quot;. If the update is done for multiple epochs, the data that is learned is outdated already after the first iteration of optimization, since the model's weights changed. In other algorithms, like A2C, there is only one optimization step done, instead of <span class="math-container">$K$</span> epochs.</p>
  <p><a href="https://i.sstatic.net/Fpcbx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Fpcbx.png" alt="enter image description here" /></a></p>
  <p>Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is an (empirical) proof that this still leads to a correct weight updating?</p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><actor-critic-methods><proximal-policy-optimization>
Title: Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm
UserDisplayName: Marcel_marcel1991
ThreadText: # Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm
  
  (Asked by: username_0 on 2018-03-04)
  
  I have a general question about the updating of the network/model in the [PPO algorithm](https://arxiv.org/abs/1707.06347).
  
  If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).
  
  Basically, in the pseudo-code of the algorithm, I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs, the data that is learned is outdated already after the first iteration of optimization, since the model's weights changed. In other algorithms, like A2C, there is only one optimization step done, instead of $K$ epochs.
  
  [![enter image description here](https://i.sstatic.net/Fpcbx.png)](https://i.sstatic.net/Fpcbx.png)
  
  Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is an (empirical) proof that this still leads to a correct weight updating?
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, actor-critic-methods, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-14. Score: 1 | Accepted answer
  
  Seems this question was asked and answered in [openai/baselines github issue](https://github.com/openai/baselines/issues/316). The issue has been closed for a while.
  
  Below is an answer provided by @matthiasplappert which has the most "thumbs up":
  
  > To clarify: PPO is an on-policy algorithm so you are correct that going over the same data multiple times is technically incorrect.
  >
  > However, we found that PPO is actually quite okay with doing this and we still get stable convergence. This is likely due to the proximal trust region constrained that we enforce, which means that the policy cannot change that much anyway when going over the current set of transitions multiple times, making it still approximately on-policy. You can of course get rid of this but then you'll need more samples.
  
  
  ---
  
  