Answers.Body: <p>The first way, as OpenAI researchers suggest also, is to use a fixed standard deviation to produce continuous actions instead of a trainable one. This value is choosen experimentally (from my experience I recommend between 1 - 1.5, but depending on your problem, other values might fit better). Neglect the 'log', just a positive value is needed (the reason there is a log is because when working with trainable std dev, the network's head produces negative values so they are considered to be log(sigma), they are exponentiated afterwards to obtain sigma and used).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-07T06:53:53.610Z
Answers.LastActivityDate: 2023-09-07T06:53:53.610Z
Answers.UserDisplayName: Formula PS4
Body: <p>I have a question about implementing policy gradient methods for problems with continuous action spaces.</p>
  <p>Assume that actions are sampled from a diagonal Gaussian distribution with mean vector <span class="math-container">$\mu$</span> and standard deviation vector <span class="math-container">$\sigma$</span>. As far as I understand, we can define a neural network that takes the current state as the input and returns a <span class="math-container">$\mu$</span> as its output. According to <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html" rel="nofollow noreferrer">OpenAI Spinning Up</a>, the standard deviation <span class="math-container">$\sigma$</span> can be represented in two different ways:</p>
  <p><a href="https://i.sstatic.net/Xkgln.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Xkgln.png" alt="enter image description here" /></a></p>
  <p>I don't completely understand the first method. Does it mean that we must set the log standard deviations to fix numbers? Then how do we choose these numbers?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-19T12:36:00.263Z
Comments.Text: If you don't get an answer meanwhile, it may be a good idea to look at existing implementations (e.g. on Github).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><policy-gradients><probability-distribution>
Title: In continuous action spaces, how is the standard deviation, associated with Gaussian distribution from which actions are sampled, represented?
UserDisplayName: M.S.
ThreadText: # In continuous action spaces, how is the standard deviation, associated with Gaussian distribution from which actions are sampled, represented?
  
  (Asked by: username_0 on 2020-07-18)
  
  I have a question about implementing policy gradient methods for problems with continuous action spaces.
  
  Assume that actions are sampled from a diagonal Gaussian distribution with mean vector $\mu$ and standard deviation vector $\sigma$. As far as I understand, we can define a neural network that takes the current state as the input and returns a $\mu$ as its output. According to [OpenAI Spinning Up](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html), the standard deviation $\sigma$ can be represented in two different ways:
  
  [![enter image description here](https://i.sstatic.net/Xkgln.png)](https://i.sstatic.net/Xkgln.png)
  
  I don't completely understand the first method. Does it mean that we must set the log standard deviations to fix numbers? Then how do we choose these numbers?
  
  **Tags:** reinforcement-learning, policy-gradients, probability-distribution
  > **Comments:**
  > **username_1**: If you don't get an answer meanwhile, it may be a good idea to look at existing implementations (e.g. on Github).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-07. Score: 0
  
  The first way, as OpenAI researchers suggest also, is to use a fixed standard deviation to produce continuous actions instead of a trainable one. This value is choosen experimentally (from my experience I recommend between 1 - 1.5, but depending on your problem, other values might fit better). Neglect the 'log', just a positive value is needed (the reason there is a log is because when working with trainable std dev, the network's head produces negative values so they are considered to be log(sigma), they are exponentiated afterwards to obtain sigma and used).
  
  
  ---
  
  