Answers.Body: <p>Using PPO from stable baselines 3, I see clearly that the variance of the actions are reducing in the environment I am using, where the optimal actions are static and not dependent on the state. This seems to suggest that the log_std parameter is being learned for the SBL3 implementation.</p>
  <p>In the docs for the SBL3-PPO, the description says log_std_init is the initial value which also suggests it is a learned parameter.</p>
  <p>log_std_init (float) – Initial value for the log standard deviation</p>
  <p>Of course, it is ultimately a design choice whether you want to make this parameter learnable.</p>
  <p>Update: Upon inspecting the PPO paper again, it is stated that they using linear annealing of the log_std parameter. However, when I inspect the log_std parameter in my experiments which start at 0 and has 3 dimensions, the values are not the same. This leads me to think there is both annealing and learning at the same time.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-18T07:15:57.253Z
Answers.LastActivityDate: 2022-09-18T07:54:04.787Z
Answers.LastEditDate: 2022-09-18T07:54:04.787Z
Answers.UserDisplayName: languageoftheuniverse
Body: <p>After having read <a href="https://link.springer.com/content/pdf/10.1007/BF00992696.pdf" rel="noreferrer">Williams (1992)</a>, where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating <em>continuous</em> output values, I assumed that this would be common practice nowadays in the domain of Deep Reinforcement Learning (DRL).
  In the <a href="http://proceedings.mlr.press/v37/schulman15.pdf" rel="noreferrer">supplementary material</a> associated with the paper introducing <a href="http://proceedings.mlr.press/v37/schulman15.pdf" rel="noreferrer">Trust Region Policy Optimization</a> (TRPO), however, it is stated that:</p>
  <blockquote>
  <p>A neural network with several fully-connected
  (dense) layers maps from the input features to the mean of a Gaussian distribution. A separate set of parameters specifies the log standard deviation of each element. More concretely, the parameters include a set of weights and biases for the neural network computing the mean, <span class="math-container">$\{W_i , b_i\}_{i=1}^L$</span> , and a vector <span class="math-container">$r$</span> (log standard deviation) with the same dimension as <span class="math-container">$a$</span>. Then, the policy is defined by the normal distribution <span class="math-container">$\mathcal{N}(\text{mean}=\text{NeuralNet}(s; \{W_i , b_i\}_{i=1}^L), \text{stdev}=\text{exp}(r))$</span>.</p>
  </blockquote>
  <p>where <span class="math-container">$s$</span> refers to a state and <span class="math-container">$a$</span> to a predicted action (respectively a vector of actions if multiple outputs are generated concurrently).</p>
  <p>To me this suggests that the standard deviation stdev (being a function of <span class="math-container">$r$</span>) is actually not learned when training a TRPO agent, but that it is solely determined by some possibly constant vector <span class="math-container">$r$</span>.</p>
  <p>Since I found the idea of adjusting both the mean and standard deviation together when training a REINFORCE agent quite reasonable, I got wondering whether it is actually true that TRPO agents do not treat the standard deviation for sampling output values as a trainable parameter, but just as a function of the state-independent vector <span class="math-container">$r$</span>.
  (Pretty much the same shall then apply to <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="noreferrer">Proximal Policy Optimization</a> (PPO) agents as well, since they are reported to follow TRPO's model architecture in the continuous output case.)</p>
  <p>In search for an answer, I browsed OpenAI's baselines repository containing reference implementations of both TRPO and PPO.
  In my understanding of their <a href="https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/distributions.py#L104" rel="noreferrer">code</a>, the code seems to confirm my assumption that standard deviation is a non-trainable parameter and that it is, instead of being trainable, taken to be a constant.</p>
  <p>Now, I was wondering whether my understanding of the procedure how TRPO (and PPO) computes standard deviation(s) is correct or whether I misunderstood or overlooked something important here.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-12T00:36:43.290Z
Comments.Text: Hi, have found a confirmation? I have the same question as yours.
Comments.UserDisplayName: kz28
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-18T09:21:27.937Z
Comments.Text: It’s a modelling choice. You can learn it, or fix it to a value of your choosing.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization><trust-region-policy-optimization>
Title: Is (log-)standard deviation learned in TRPO and PPO or fixed instead?
UserDisplayName: Daniel B.
ThreadText: # Is (log-)standard deviation learned in TRPO and PPO or fixed instead?
  
  (Asked by: username_0 on 2021-02-12)
  
  After having read [Williams (1992)](https://link.springer.com/content/pdf/10.1007/BF00992696.pdf), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating *continuous* output values, I assumed that this would be common practice nowadays in the domain of Deep Reinforcement Learning (DRL).
  In the [supplementary material](http://proceedings.mlr.press/v37/schulman15.pdf) associated with the paper introducing [Trust Region Policy Optimization](http://proceedings.mlr.press/v37/schulman15.pdf) (TRPO), however, it is stated that:
  
  > A neural network with several fully-connected
  > (dense) layers maps from the input features to the mean of a Gaussian distribution. A separate set of parameters specifies the log standard deviation of each element. More concretely, the parameters include a set of weights and biases for the neural network computing the mean, $\{W\_i , b\_i\}\_{i=1}^L$ , and a vector $r$ (log standard deviation) with the same dimension as $a$. Then, the policy is defined by the normal distribution $\mathcal{N}(\text{mean}=\text{NeuralNet}(s; \{W\_i , b\_i\}\_{i=1}^L), \text{stdev}=\text{exp}(r))$.
  
  where $s$ refers to a state and $a$ to a predicted action (respectively a vector of actions if multiple outputs are generated concurrently).
  
  To me this suggests that the standard deviation stdev (being a function of $r$) is actually not learned when training a TRPO agent, but that it is solely determined by some possibly constant vector $r$.
  
  Since I found the idea of adjusting both the mean and standard deviation together when training a REINFORCE agent quite reasonable, I got wondering whether it is actually true that TRPO agents do not treat the standard deviation for sampling output values as a trainable parameter, but just as a function of the state-independent vector $r$.
  (Pretty much the same shall then apply to [Proximal Policy Optimization](https://arxiv.org/pdf/1707.06347.pdf) (PPO) agents as well, since they are reported to follow TRPO's model architecture in the continuous output case.)
  
  In search for an answer, I browsed OpenAI's baselines repository containing reference implementations of both TRPO and PPO.
  In my understanding of their [code](https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/common/distributions.py#L104), the code seems to confirm my assumption that standard deviation is a non-trainable parameter and that it is, instead of being trainable, taken to be a constant.
  
  Now, I was wondering whether my understanding of the procedure how TRPO (and PPO) computes standard deviation(s) is correct or whether I misunderstood or overlooked something important here.
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization, trust-region-policy-optimization
  
  > **Comments:**
  > **username_1**: Hi, have found a confirmation? I have the same question as yours.  
  > **username_2**: It’s a modelling choice. You can learn it, or fix it to a value of your choosing.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-09-18. Score: 1
  
  Using PPO from stable baselines 3, I see clearly that the variance of the actions are reducing in the environment I am using, where the optimal actions are static and not dependent on the state. This seems to suggest that the log\_std parameter is being learned for the SBL3 implementation.
  
  In the docs for the SBL3-PPO, the description says log\_std\_init is the initial value which also suggests it is a learned parameter.
  
  log\_std\_init (float) – Initial value for the log standard deviation
  
  Of course, it is ultimately a design choice whether you want to make this parameter learnable.
  
  Update: Upon inspecting the PPO paper again, it is stated that they using linear annealing of the log\_std parameter. However, when I inspect the log\_std parameter in my experiments which start at 0 and has 3 dimensions, the values are not the same. This leads me to think there is both annealing and learning at the same time.
  
  
  ---
  
  