Answers.Body: <p>Both implementations may be closer than you think.</p>
  
  <p><strong>In short:</strong></p>
  
  <p><strong>PPO has both parts</strong>: there is <strong>noisiness in draws during training</strong> (with learned standard deviation), helping to explore new promising actions/policies. And there is a <strong>term added to the loss function aiming to prevent a collapse of the noisiness</strong>, to help ensure exploration continues and we don't get stuck at a bad (local) equilibrium.</p>
  
  <p>In fact, for continuous action, the term for <strong>entropy in the loss function</strong> you describe in Ex. 2, can <strong>make sense only when the actions are stochastic</strong>, i.e. when there action choice has some standard deviation the way you describe in Ex. 1.</p>
  
  <p><strong>More detail:</strong></p>
  
  <p>On one hand, PPO (at least for continuous action), trains a central/deterministic value (say the mean policy or close to mean) targeting a most profitable action path. On the other hand, along with it, a standard deviation making the actions a random draw with noise around the deterministic value. This is the part you describe in <em>Example 1</em>. The noise helps explore new paths and to update the policy according to the rewards on these sampled paths.
  Entropy itself is a measure of the noisiness of the draws, an thus also an indirect indicator for the trained standard deviation value(s) of the policy.</p>
  
  <p>Now, entropy tends to decay as training progresses, that is, the random draws become progressively less random. This can be good for reward maximization - really the best draws are taken for reward maximization - but it is bad for further improvements of policy: improvement may halt or slow down as exploration of new action paths fades.</p>
  
  <p>This is where entropy <em>encouragement</em> comes in. PPO foresees the inclusion of entropy in the loss function: we reduce the loss by <code>x * entropy</code>, with <code>x</code> the entropy coefficient (e.g. 0.01), incentivizing the learning network to increase the standard deviations (or, to not let them drop too much). This part is what you describe in <em>Example 2</em>.</p>
  
  <p><strong>Further notes:</strong></p>
  
  <ul>
  <li><p>During exploitation, we'd typically turn off the noise (implicitly assuming <code>action std = 0</code>) and pick deterministic actions: in normal cases this increases the payoffs; we're choosing our best action estimate, rather than at a random value around it.</p></li>
  <li><p>People are not always precise when referring to the model's <code>entropy</code> vs. the <code>entropy coefficient</code> added to the loss function.</p></li>
  <li><p>Other RL algorithms with continuous action tend to use noisy drwas with standard deviations/entropy too.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T20:38:53.007Z
Answers.Text: the entropy of a constant distribution is zero.
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-30T08:21:56.490Z
Answers.LastActivityDate: 2020-04-30T10:42:27.763Z
Answers.LastEditDate: 2020-04-30T10:42:27.763Z
Answers.UserDisplayName: FlorianH
Body: <p>When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations :</p>
  
  <h3>Exploration with std</h3>
  
  <ol>
  <li>Collect trajectories on <span class="math-container">$N$</span> timesteps, by using a policy-centered distribution with progressively trained std variable for exploration</li>
  <li>Train policy function on <span class="math-container">$K$</span> steps</li>
  <li>Train value function on <span class="math-container">$K$</span> steps</li>
  </ol>
  
  <p>For example, the <a href="https://spinningup.openai.com/en/latest/algorithms/ppo.html" rel="nofollow noreferrer">OpenAI's implementation</a>.</p>
  
  <h3>Exploration with entropy</h3>
  
  <ol>
  <li>Collect trajectories on <span class="math-container">$N$</span> timesteps, by using policy function directly</li>
  <li>Train policy and value function at the same time on <span class="math-container">$K$</span> steps, with a common loss for the two models, with <strong>additional entropy</strong> bonus for exploration purpose.</li>
  </ol>
  
  <p>For example, the PPO algorithm as described in the <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">official paper</a>.</p>
  
  <p>What are the pros/cons of these two algorithms?</p>
  
  <p>Is this specific to PPO, or is this a classic question concerning policy gradients algorithms, in general?</p>
  
Tags: <neural-networks><reinforcement-learning><policy-gradients><proximal-policy-optimization>
Title: What are the pros and cons of using standard deviation or entropy for exploration in PPO?
UserDisplayName: Loheek
ThreadText: # What are the pros and cons of using standard deviation or entropy for exploration in PPO?
  
  (Asked by: username_0 on 2019-09-08)
  
  When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations :
  
  ### Exploration with std
  
  1. Collect trajectories on $N$ timesteps, by using a policy-centered distribution with progressively trained std variable for exploration
  2. Train policy function on $K$ steps
  3. Train value function on $K$ steps
  
  For example, the [OpenAI's implementation](https://spinningup.openai.com/en/latest/algorithms/ppo.html).
  
  ### Exploration with entropy
  
  1. Collect trajectories on $N$ timesteps, by using policy function directly
  2. Train policy and value function at the same time on $K$ steps, with a common loss for the two models, with **additional entropy** bonus for exploration purpose.
  
  For example, the PPO algorithm as described in the [official paper](https://arxiv.org/pdf/1707.06347.pdf).
  
  What are the pros/cons of these two algorithms?
  
  Is this specific to PPO, or is this a classic question concerning policy gradients algorithms, in general?
  
  **Tags:** neural-networks, reinforcement-learning, policy-gradients, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-30. Score: 4
  
  Both implementations may be closer than you think.
  
  **In short:**
  
  **PPO has both parts**: there is **noisiness in draws during training** (with learned standard deviation), helping to explore new promising actions/policies. And there is a **term added to the loss function aiming to prevent a collapse of the noisiness**, to help ensure exploration continues and we don't get stuck at a bad (local) equilibrium.
  
  In fact, for continuous action, the term for **entropy in the loss function** you describe in Ex. 2, can **make sense only when the actions are stochastic**, i.e. when there action choice has some standard deviation the way you describe in Ex. 1.
  
  **More detail:**
  
  On one hand, PPO (at least for continuous action), trains a central/deterministic value (say the mean policy or close to mean) targeting a most profitable action path. On the other hand, along with it, a standard deviation making the actions a random draw with noise around the deterministic value. This is the part you describe in *Example 1*. The noise helps explore new paths and to update the policy according to the rewards on these sampled paths.
  Entropy itself is a measure of the noisiness of the draws, an thus also an indirect indicator for the trained standard deviation value(s) of the policy.
  
  Now, entropy tends to decay as training progresses, that is, the random draws become progressively less random. This can be good for reward maximization - really the best draws are taken for reward maximization - but it is bad for further improvements of policy: improvement may halt or slow down as exploration of new action paths fades.
  
  This is where entropy *encouragement* comes in. PPO foresees the inclusion of entropy in the loss function: we reduce the loss by `x * entropy`, with `x` the entropy coefficient (e.g. 0.01), incentivizing the learning network to increase the standard deviations (or, to not let them drop too much). This part is what you describe in *Example 2*.
  
  **Further notes:**
  
  * During exploitation, we'd typically turn off the noise (implicitly assuming `action std = 0`) and pick deterministic actions: in normal cases this increases the payoffs; we're choosing our best action estimate, rather than at a random value around it.
  * People are not always precise when referring to the model's `entropy` vs. the `entropy coefficient` added to the loss function.
  * Other RL algorithms with continuous action tend to use noisy drwas with standard deviations/entropy too.
  
  > **Comments:**
  > **username_2**: the entropy of a constant distribution is zero.  
  
  ---
  
  