Answers.Body: <p><strong>Loss</strong>. In the context of Deep Learning and Deep Reinforcement Learning, &quot;training&quot; is just a fancy word for &quot;optimization&quot;. You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to <em>maximize</em> the expected <em>return</em> - while most of the Deep Learning classification problems are <em>minimizing</em> some <em>loss</em>.</p>
  <p>Most of the modern Deep Learning programming frameworks was created with the classification problems in mind, so they call the optimization goals &quot;losses&quot;: <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses" rel="nofollow noreferrer">tf.keras</a>, <a href="https://pytorch.org/docs/stable/nn.html#loss-functions" rel="nofollow noreferrer">pytorch</a>. The good news is that you can easily convert return-maximization problem to loss-minimization problem by just changing the sign of your target function.</p>
  <p>Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign.</p>
  <p><strong>Value Function</strong>. Given an agent's state, Reinforcement Learning tries to find the best agent's action that maximized the expected return. The problem is that the expected return is also unknown. Note on the difference between &quot;reward&quot; and &quot;return&quot;. Rewards <span class="math-container">$r_t$</span> are immediate bits of positive/negative reinforcement that the agent receives obtain after taking an action. The &quot;return&quot; <span class="math-container">$G_t$</span> is the function of a particular trajectory - it is the discounted sum of future rewards starting from step <span class="math-container">$t$</span>:</p>
  <p><span class="math-container">$$G_t = \sum_k \gamma^k r_{t+k+1}$$</span></p>
  <p>The expectation of the return at state <span class="math-container">$s$</span> given a policy <span class="math-container">$\pi$</span> is called value <span class="math-container">$V^\pi(s)$</span>:</p>
  <p><span class="math-container">$$V^\pi(s) = \mathbb{E}_{a\sim\pi}\left[ G_t\right|\left. s_t=s\right]$$</span></p>
  <p>In principle, it is possible to estimate <span class="math-container">$V^\pi(s)$</span> by doing Monte Carlo: start from the state <span class="math-container">$s$</span> and run many simulations following <span class="math-container">$\pi$</span> and average all the returns you've got. For most practical cases this approach is infeasible. Modern Deep Reinforcement Learning algorithms try to reduce computational load by using neural networks to approximate <span class="math-container">$V(s)$</span>.</p>
  <p>Having the value function approximation <span class="math-container">$V(s)$</span>, one can then try to optimize the policy approximation function <span class="math-container">$\pi(a|s)$</span> using the policy gradient update. Such setup: with one approximator for policy and another approximator for value function is called an &quot;Actor-Critic&quot; family of Reinforcement Learning algorithms.</p>
  <p>BTW, it doesn't look like your value function &quot;stabilizes&quot; during training. And it also doesn't change too much if you look at the vertical scale of your plot.</p>
  <p><strong>Surrogate Loss</strong>. In practice, the policy gradient optimization step above suffers from instabilities. The gradient step tends to change the policy too strongly, which makes the approximation error of the value function too large.</p>
  <p>To deal with this, the TRPO/PPO algorithms introduce a kind of clipping of the loss gradient that constrains the policy gradient changes from being too large. The new function that produces these clipped/constrained gradients is called a &quot;surrogate loss&quot;.</p>
  <p><strong>mean_noise_std</strong>. This doesn't look like a standard name for anything, but I suspect that this is a version of PPO with <a href="https://openai.com/blog/better-exploration-with-parameter-noise/" rel="nofollow noreferrer">artificial noise</a> added for better exploration of the agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T16:35:57.753Z
Answers.Text: Thank you for the kind explanation. "The problem is that the expected reward is also unknown" - why is this the case? We have physic simulator. We can just simulate the robot's behavior with the given agent's state.
Answers.UserDisplayName: Joe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T16:39:46.367Z
Answers.Text: "Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign." - I have added plots of an ideal training process. As you can see, as the reward gets maximized around 1k iteration, the loss also gets maximized. This behavior somehow doesn't fit your explanation... Could you elaborate a bit more?
Answers.UserDisplayName: Joe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T19:37:01.680Z
Answers.Text: @Joe First of all, I've made a slight terminological mistake mixing up "reward" and "return".  I've tried to expand on your first comment - In principle, you can estimate $V(s)$ by running simulations of the robot's behavior from the given agent's state. Say you choose to run it 100 times per state. And you have 10000 trajectories. And each trajectory has 10000 states in it....
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T19:40:19.223Z
Answers.Text: @Joe I cannot comment on your "ideal training"plots apart from the fact that you have "reward" here instead of "loss".  I'll need more information on how you got it and why is it "ideal". Are you trying to do some online course and that's an expected behavior?
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T20:03:18.087Z
Answers.Text: I am using Nvidia Isaac Gym and the later plots are from a given example from Nvidia thats why I called it ideal. In the example, the robot also manages to reach the goal position about after 1k iteration.
Answers.UserDisplayName: Joe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T14:17:05.773Z
Answers.LastActivityDate: 2021-05-15T19:34:16.453Z
Answers.LastEditDate: 2021-05-15T19:34:16.453Z
Answers.UserDisplayName: Kostya
Body: <p>I am currently using the PPO method from the <a href="https://developer.nvidia.com/isaac-gym" rel="nofollow noreferrer">Nvidia's Isaac gym</a> to train an agent for my robot. Below, you can see the plot which corresponds to a training process.</p>
  <p>I know that something is massively wrong with my training process (since the robot does not manage to get a nice policy), so I am trying to understand the training process more with the help of the below values being logged during the problematic training.</p>
  <p><a href="https://i.sstatic.net/OcwGx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OcwGx.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/n8zT4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/n8zT4.png" alt="enter image description here" /></a></p>
  <p>So far, I could find out what this value function means: the reward function stabilizes, thus the loss value function also stabilizes, which means that my robot should explore more to fail and learn from the fail!</p>
  <p>But what about the other two plots, <code>surrogate</code> and <code>mean_noise_std</code>? How should one interpret those values?</p>
  <hr />
  <p>The ideal training process</p>
  <p><a href="https://i.sstatic.net/atzVC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/atzVC.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/7byOv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7byOv.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/1jX4G.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1jX4G.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?
UserDisplayName: Joe
ThreadText: # How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?
  
  (Asked by: username_0 on 2021-05-15)
  
  I am currently using the PPO method from the [Nvidia's Isaac gym](https://developer.nvidia.com/isaac-gym) to train an agent for my robot. Below, you can see the plot which corresponds to a training process.
  
  I know that something is massively wrong with my training process (since the robot does not manage to get a nice policy), so I am trying to understand the training process more with the help of the below values being logged during the problematic training.
  
  [![enter image description here](https://i.sstatic.net/OcwGx.png)](https://i.sstatic.net/OcwGx.png)
  
  [![enter image description here](https://i.sstatic.net/n8zT4.png)](https://i.sstatic.net/n8zT4.png)
  
  So far, I could find out what this value function means: the reward function stabilizes, thus the loss value function also stabilizes, which means that my robot should explore more to fail and learn from the fail!
  
  But what about the other two plots, `surrogate` and `mean_noise_std`? How should one interpret those values?
  
  ---
  
  The ideal training process
  
  [![enter image description here](https://i.sstatic.net/atzVC.png)](https://i.sstatic.net/atzVC.png)
  [![enter image description here](https://i.sstatic.net/7byOv.png)](https://i.sstatic.net/7byOv.png)
  [![enter image description here](https://i.sstatic.net/1jX4G.png)](https://i.sstatic.net/1jX4G.png)
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-15. Score: 2
  
  **Loss**. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to *maximize* the expected *return* - while most of the Deep Learning classification problems are *minimizing* some *loss*.
  
  Most of the modern Deep Learning programming frameworks was created with the classification problems in mind, so they call the optimization goals "losses": [tf.keras](https://www.tensorflow.org/api_docs/python/tf/keras/losses), [pytorch](https://pytorch.org/docs/stable/nn.html#loss-functions). The good news is that you can easily convert return-maximization problem to loss-minimization problem by just changing the sign of your target function.
  
  Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign.
  
  **Value Function**. Given an agent's state, Reinforcement Learning tries to find the best agent's action that maximized the expected return. The problem is that the expected return is also unknown. Note on the difference between "reward" and "return". Rewards $r\_t$ are immediate bits of positive/negative reinforcement that the agent receives obtain after taking an action. The "return" $G\_t$ is the function of a particular trajectory - it is the discounted sum of future rewards starting from step $t$:
  
  $$G\_t = \sum\_k \gamma^k r\_{t+k+1}$$
  
  The expectation of the return at state $s$ given a policy $\pi$ is called value $V^\pi(s)$:
  
  $$V^\pi(s) = \mathbb{E}\_{a\sim\pi}\left[ G\_t\right|\left. s\_t=s\right]$$
  
  In principle, it is possible to estimate $V^\pi(s)$ by doing Monte Carlo: start from the state $s$ and run many simulations following $\pi$ and average all the returns you've got. For most practical cases this approach is infeasible. Modern Deep Reinforcement Learning algorithms try to reduce computational load by using neural networks to approximate $V(s)$.
  
  Having the value function approximation $V(s)$, one can then try to optimize the policy approximation function $\pi(a|s)$ using the policy gradient update. Such setup: with one approximator for policy and another approximator for value function is called an "Actor-Critic" family of Reinforcement Learning algorithms.
  
  BTW, it doesn't look like your value function "stabilizes" during training. And it also doesn't change too much if you look at the vertical scale of your plot.
  
  **Surrogate Loss**. In practice, the policy gradient optimization step above suffers from instabilities. The gradient step tends to change the policy too strongly, which makes the approximation error of the value function too large.
  
  To deal with this, the TRPO/PPO algorithms introduce a kind of clipping of the loss gradient that constrains the policy gradient changes from being too large. The new function that produces these clipped/constrained gradients is called a "surrogate loss".
  
  **mean\_noise\_std**. This doesn't look like a standard name for anything, but I suspect that this is a version of PPO with [artificial noise](https://openai.com/blog/better-exploration-with-parameter-noise/) added for better exploration of the agent.
  
  > **Comments:**
  > **username_0**: Thank you for the kind explanation. "The problem is that the expected reward is also unknown" - why is this the case? We have physic simulator. We can just simulate the robot's behavior with the given agent's state.  
  > **username_0**: "Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign." - I have added plots of an ideal training process. As you can see, as the reward gets maximized around 1k iteration, the loss also gets maximized. This behavior somehow doesn't fit your explanation... Could you elaborate a bit more?  
  > **username_1**: @username_0 First of all, I've made a slight terminological mistake mixing up "reward" and "return".  I've tried to expand on your first comment - In principle, you can estimate $V(s)$ by running simulations of the robot's behavior from the given agent's state. Say you choose to run it 100 times per state. And you have 10000 trajectories. And each trajectory has 10000 states in it....  
  > **username_1**: @username_0 I cannot comment on your "ideal training"plots apart from the fact that you have "reward" here instead of "loss".  I'll need more information on how you got it and why is it "ideal". Are you trying to do some online course and that's an expected behavior?  
  > **username_0**: I am using Nvidia Isaac Gym and the later plots are from a given example from Nvidia thats why I called it ideal. In the example, the robot also manages to reach the goal position about after 1k iteration.  
  
  ---
  
  