Answers.Body: <blockquote>
  <p>I didn't quite understand how the PPO updates the network, according to what?</p>
  </blockquote>
  <p>PPO implements what is called a <em>policy gradient</em> algorithm. What policy gradient is essentially doing is updating the policy in the direction of better return (return is the discounted sum of rewards during an episode).
  Let's say the policy is parametrized by neural network weights <span class="math-container">$\phi$</span>.
  Then the weights update in the most basic policy gradient algorithm is computed as:</p>
  <p><span class="math-container">$$\phi = \phi + \alpha \nabla \log \pi_{\phi}(a_t,s_t) \cdot V_t,$$</span></p>
  <p>where  <span class="math-container">$V_t$</span> is the return. In case of PPO, <span class="math-container">$V_t$</span> is actually estimated (generalized) advantage function implemented by another neural network. This second neural network is often called <em>critic</em>, while your policy network is called an <em>actor</em>. Meaning that the actor is taking decisions, while the critic helps to train the actor by providing &quot;an advice&quot; in which direction to move.</p>
  <blockquote>
  <p>There needs to be target values right? Or at least a loss function?</p>
  </blockquote>
  <p>The loss function in PPO is</p>
  <p><span class="math-container">$$L^{PPO} = L^{PG} + c_1 L^{VF} - c_2 S,$$</span></p>
  <p>where <span class="math-container">$L^{PG}$</span> is policy gradient loss calculated for actor performance, i.e. how good were actor's actions. In a simple policy gradient algorithm it would be <span class="math-container">$L^{PG} = -\sum_{t} \log \pi_{\phi}(a_t|s_t) \cdot V_t$</span>, while in PPO it is a bit more sophisticated</p>
  <p><span class="math-container">$$ L^{PG}(\phi) = \sum_t \max \left( -r_t(\phi) \hat A_t, -\text{clip}(r_t(\phi), 1 - \epsilon, 1 + \epsilon) \hat A_t \right),$$</span></p>
  <p>where <span class="math-container">$\hat A_t$</span> is empirical advantage function (the same as <span class="math-container">$V_t$</span> before) and <span class="math-container">$r_t(\phi) = \frac{\pi_{\phi}(a_t|s_t)}{\pi_{\phi_{\text{old}}}(a_t|s_t)}$</span> is the probability ratio of how better is the new policy is compared to the old one.</p>
  <p><span class="math-container">$L^{VF} = (V_{\theta}(s_t) - V_t^{\text{targ}})^2$</span> is value function loss function (<span class="math-container">$\theta$</span> are critic network parameters). This one is calculated for critic performance, i.e. how good it is able to estimate advantages. And
  <span class="math-container">$S$</span> is what is called &quot;entropy&quot; bonus (to increase the entropy, i.e. to stimulate exploration).</p>
  <blockquote>
  <p>Should I treat it like a target values of the network?</p>
  </blockquote>
  <p>No, not necessary. Note also that unlike DQN, PPO is an on-policy algorithm, which means that it throws away the episode information (states, rewards, actions) as soon as it is done with it.</p>
  <p>For more details, see this <a href="https://ppo-details.cleanrl.dev/2021/11/05/ppo-implementation-details/" rel="nofollow noreferrer">excellent post</a> about PPO details. There are also <a href="https://www.youtube.com/watch?v=MEt6rrxH8W4&amp;list=PLD80i8An1OEHhcxclwq8jOMam0m0M9dQ_&amp;index=2" rel="nofollow noreferrer">three</a> <a href="https://www.youtube.com/watch?v=05RMTj-2K_Y&amp;list=PLD80i8An1OEHhcxclwq8jOMam0m0M9dQ_&amp;index=3" rel="nofollow noreferrer">Youtube</a> <a href="https://www.youtube.com/watch?v=BvZvx7ENZBw&amp;list=PLD80i8An1OEHhcxclwq8jOMam0m0M9dQ_&amp;index=3" rel="nofollow noreferrer">videos</a> explaining the subject. And I also wrote a <a href="https://penkovsky.com/neural-networks/beyond/" rel="nofollow noreferrer">post implementing</a> PPO from scratch (first using REINFORCE to illustrate the simplest policy gradient algorithm). Also don't hesitate to read the original <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">PPO paper</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T12:20:56.157Z
Answers.Text: Thanks for the detailed answer and sources. I will check them. Just one more question: Is the critic something like target network in DQN?
Answers.UserDisplayName: Ege
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T12:23:29.513Z
Answers.Text: No, critic is a separate network, which outputs a scalar value (estimated advantage in case of PPO). Whereas a target network is a clone of your actor network in DQN.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T12:26:36.293Z
Answers.Text: This lecture by David Silver will provide the necessary background about actor-critic architecture https://www.youtube.com/watch?v=KHZVXao4qXs
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-14T22:47:57.897Z
Answers.Text: _No, critic is a separate network_ not always, it definitely can be the same network with 2 output heads, which for example has been done with AlphaGo
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-15T08:58:43.293Z
Answers.Text: Indeed, to be technically more precise, you can have a network with two outputs. The question was if critic is a target network. And it is not.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T12:00:39.200Z
Answers.LastActivityDate: 2023-08-14T07:04:05.103Z
Answers.LastEditDate: 2023-08-14T07:04:05.103Z
Answers.UserDisplayName: penkovsky
Body: <p>I am trying to implement PPO to my reinforcement agents. I have a classic neural network that represents the policy. I didn't quite understand how the PPO updates the network, according to what? There needs to be target values right? Or at least a loss function? For example in DQN a second network gives target Q values and the backpropagation is done according to that values for example with MSE loss function. But I don't know where I should put the returned value from PPO function. I don't get it. How can I backpropagate according to PPO values? Should I treat it like a target values of the network? If I should not, what are the target values or what is the loss?</p>
  
Tags: <neural-networks><reinforcement-learning><proximal-policy-optimization>
Title: How Can I Backpropagate My Network with PPO
UserDisplayName: Ege
ThreadText: # How Can I Backpropagate My Network with PPO
  
  (Asked by: username_0 on 2023-08-13)
  
  I am trying to implement PPO to my reinforcement agents. I have a classic neural network that represents the policy. I didn't quite understand how the PPO updates the network, according to what? There needs to be target values right? Or at least a loss function? For example in DQN a second network gives target Q values and the backpropagation is done according to that values for example with MSE loss function. But I don't know where I should put the returned value from PPO function. I don't get it. How can I backpropagate according to PPO values? Should I treat it like a target values of the network? If I should not, what are the target values or what is the loss?
  
  **Tags:** neural-networks, reinforcement-learning, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-13. Score: 3 | Accepted answer
  
  > I didn't quite understand how the PPO updates the network, according to what?
  
  PPO implements what is called a *policy gradient* algorithm. What policy gradient is essentially doing is updating the policy in the direction of better return (return is the discounted sum of rewards during an episode).
  Let's say the policy is parametrized by neural network weights $\phi$.
  Then the weights update in the most basic policy gradient algorithm is computed as:
  
  $$\phi = \phi + \alpha \nabla \log \pi\_{\phi}(a\_t,s\_t) \cdot V\_t,$$
  
  where $V\_t$ is the return. In case of PPO, $V\_t$ is actually estimated (generalized) advantage function implemented by another neural network. This second neural network is often called *critic*, while your policy network is called an *actor*. Meaning that the actor is taking decisions, while the critic helps to train the actor by providing "an advice" in which direction to move.
  
  > There needs to be target values right? Or at least a loss function?
  
  The loss function in PPO is
  
  $$L^{PPO} = L^{PG} + c\_1 L^{VF} - c\_2 S,$$
  
  where $L^{PG}$ is policy gradient loss calculated for actor performance, i.e. how good were actor's actions. In a simple policy gradient algorithm it would be $L^{PG} = -\sum\_{t} \log \pi\_{\phi}(a\_t|s\_t) \cdot V\_t$, while in PPO it is a bit more sophisticated
  
  $$ L^{PG}(\phi) = \sum\_t \max \left( -r\_t(\phi) \hat A\_t, -\text{clip}(r\_t(\phi), 1 - \epsilon, 1 + \epsilon) \hat A\_t \right),$$
  
  where $\hat A\_t$ is empirical advantage function (the same as $V\_t$ before) and $r\_t(\phi) = \frac{\pi\_{\phi}(a\_t|s\_t)}{\pi\_{\phi\_{\text{old}}}(a\_t|s\_t)}$ is the probability ratio of how better is the new policy is compared to the old one.
  
  $L^{VF} = (V\_{\theta}(s\_t) - V\_t^{\text{targ}})^2$ is value function loss function ($\theta$ are critic network parameters). This one is calculated for critic performance, i.e. how good it is able to estimate advantages. And
  $S$ is what is called "entropy" bonus (to increase the entropy, i.e. to stimulate exploration).
  
  > Should I treat it like a target values of the network?
  
  No, not necessary. Note also that unlike DQN, PPO is an on-policy algorithm, which means that it throws away the episode information (states, rewards, actions) as soon as it is done with it.
  
  For more details, see this [excellent post](https://ppo-details.cleanrl.dev/2021/11/05/ppo-implementation-details/) about PPO details. There are also [three](https://www.youtube.com/watch?v=MEt6rrxH8W4&list=PLD80i8An1OEHhcxclwq8jOMam0m0M9dQ_&index=2) [Youtube](https://www.youtube.com/watch?v=05RMTj-2K_Y&list=PLD80i8An1OEHhcxclwq8jOMam0m0M9dQ_&index=3) [videos](https://www.youtube.com/watch?v=BvZvx7ENZBw&list=PLD80i8An1OEHhcxclwq8jOMam0m0M9dQ_&index=3) explaining the subject. And I also wrote a [post implementing](https://username_1.com/neural-networks/beyond/) PPO from scratch (first using REINFORCE to illustrate the simplest policy gradient algorithm). Also don't hesitate to read the original [PPO paper](https://arxiv.org/pdf/1707.06347.pdf).
  
  > **Comments:**
  > **username_0**: Thanks for the detailed answer and sources. I will check them. Just one more question: Is the critic something like target network in DQN?  
  > **username_1**: No, critic is a separate network, which outputs a scalar value (estimated advantage in case of PPO). Whereas a target network is a clone of your actor network in DQN.  
  > **username_1**: This lecture by David Silver will provide the necessary background about actor-critic architecture https://www.youtube.com/watch?v=KHZVXao4qXs  
  > **username_2**: _No, critic is a separate network_ not always, it definitely can be the same network with 2 output heads, which for example has been done with AlphaGo  
  > **username_1**: Indeed, to be technically more precise, you can have a network with two outputs. The question was if critic is a target network. And it is not.  
  
  ---
  
  