Answers.Body: <p>You're right, the first time you run it the two policies (<span class="math-container">$\pi_{\theta old}$</span> and <span class="math-container">$\pi_\theta$</span>) will be the same. This means your loss is simply the advantage (since you multiply the the ratio (<span class="math-container">$r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$</span>) by the advantage (so <span class="math-container">$loss=-r_t(\theta)A_t$</span>).</p>
  
  <p>However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time <span class="math-container">$\pi_{\theta old}$</span> is different to <span class="math-container">$\pi_\theta$</span>.</p>
  
  <p>Here's a great explanation of the algorithm: <a href="https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl">https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-03T02:51:37.303Z
Answers.LastActivityDate: 2018-10-03T02:51:37.303Z
Answers.UserDisplayName: Omegastick
Body: <p>I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.</p>
  
  <p>In the paper <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">Proximal Policy Optimization Algorithms</a> (at page 5), the pseudocode of the PPO algorithm is shown:</p>
  
  <p><a href="https://i.sstatic.net/bVkQH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bVkQH.png" alt="enter image description here"></a>  </p>
  
  <p>It says to run <span class="math-container">$\pi_{\theta_{\text{old}}}$</span>, compute advantage estimates and optimize the objective. But how can we calculate <span class="math-container">$\pi_\theta$</span> for the objective ratio, since we have not updated the <span class="math-container">$\pi_{\theta_{\text{old}}}$</span> yet?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><proximal-policy-optimization>
Title: How do I calculate the policy in the Proximal Policy Optimization algorithm?
UserDisplayName: Starman
ThreadText: # How do I calculate the policy in the Proximal Policy Optimization algorithm?
  
  (Asked by: username_0 on 2018-10-02)
  
  I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.
  
  In the paper [Proximal Policy Optimization Algorithms](https://arxiv.org/pdf/1707.06347.pdf) (at page 5), the pseudocode of the PPO algorithm is shown:
  
  [![enter image description here](https://i.sstatic.net/bVkQH.png)](https://i.sstatic.net/bVkQH.png)
  
  It says to run $\pi\_{\theta\_{\text{old}}}$, compute advantage estimates and optimize the objective. But how can we calculate $\pi\_\theta$ for the objective ratio, since we have not updated the $\pi\_{\theta\_{\text{old}}}$ yet?
  
  **Tags:** reinforcement-learning, actor-critic-methods, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-03. Score: 2
  
  You're right, the first time you run it the two policies ($\pi\_{\theta old}$ and $\pi\_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi\_\theta(a|s)\over\pi\_{\theta old}(a|s)}$) by the advantage (so $loss=-r\_t(\theta)A\_t$).
  
  However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time $\pi\_{\theta old}$ is different to $\pi\_\theta$.
  
  Here's a great explanation of the algorithm: <https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl>
  
  
  ---
  
  