Answers.Body: <p>I don't really understand the confusion, as both terms are scalars (where <span class="math-container">$u_i$</span> are the components of <span class="math-container">$u$</span>).</p>
  <p>So if <span class="math-container">$u\sim \mathcal{N}(\mu, \Sigma)$</span>, then the first term would be</p>
  <p><span class="math-container">$$\log \mu(u\mid s)=-\frac{D}{2}\log(2\pi)-\frac{1}{2}\log\vert \Sigma\vert - \frac{1}{2}(u-\mu)^T\Sigma^{-1}(u-\mu) $$</span></p>
  <p>However, SAC assumes that <span class="math-container">$\Sigma$</span> is diagonal, which makes the expression easier, that is</p>
  <p><span class="math-container">$$\log \pi(a\mid s)=-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum_{i=1}^D\log\sigma_i - \frac{1}{2}\sum_{i=1}^D\frac{(u_i-\mu_i)^2}{\sigma_i}-\sum_{i=1}^D\log(1-\tanh ^2u_i) $$</span></p>
  <p>If there is a correlation between the individual actions, you would have to implement the first expression.</p>
  <p>An explicit implementation I have come across in Pytorch is the following: (where <span class="math-container">$\sqrt {2\pi} \approx2.5066$</span>)</p>
  <pre><code>sampled_u = self.sample_z(mua_tensor.data, siga_tensor.data).data
  sampled_a = torch.tanh(sampled_u)
  
  log_pi_exp = torch.sum(- (mua_tensor.data - sampled_u.data).pow(2)
                         / (siga_tensor.data.pow(2)) / 2
                         - torch.log(siga_tensor.data * torch.tensor(2.5066)),
                         dim=-1, keepdim=True)
  log_pi_exp -= torch.sum(torch.log(1.0 - sampled_a.pow(2) + EPS), dim=-1, keepdim=True)
  
  </code></pre>
  <p>However, this is not efficient and there are undoubtedly specific functions in Tensorflow that make this implicitly, but I hope this provides more intuition.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T14:48:29.233Z
Answers.Text: You are right, I got confused with the second term of the equation. Thanks for your explanation!
Answers.UserDisplayName: Luc-dotcom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-30T14:06:05.573Z
Answers.LastActivityDate: 2022-07-30T14:12:18.900Z
Answers.LastEditDate: 2022-07-30T14:12:18.900Z
Answers.UserDisplayName: matorbi
Body: <p>I'm trying to solve <strong>LunarLanderContinuous-v2</strong> (<a href="https://www.gymlibrary.ml/environments/box2d/lunar_lander/" rel="nofollow noreferrer">https://www.gymlibrary.ml/environments/box2d/lunar_lander/</a>) using <strong>Soft Actor-Critic</strong> algorithm (following the pseudocode above)</p>
  <p><img src="https://spinningup.openai.com/en/latest/_images/math/c01f4994ae4aacf299a6b3ceceedfe0a14d4b874.svg" alt="Text" /></p>
  <p>To update the actor policy (step 14) I understood that I need to:</p>
  <ol>
  <li>sample an action from a normal distribution using mean and variance that are the output of the policy network</li>
  <li>squash the action with tanh to have action bounds [-1, 1]</li>
  <li>compute the log_prob, the log probability density of the sample</li>
  <li>do the following operation in tensorflow</li>
  </ol>
  <pre><code># STEP 14
  bounded_actions_sample, predicted_next_actions_sample = policy_network.predict_actions(batch_observation)
  pdf = get_pdf(predicted_next_actions_sample, batch_observation)
  q_min = tf.math.minimum(q_network_1(batch_observation, bounded_actions_sample), q_network_2(batch_observation, bounded_actions_sample))
          
  loss_policy_network = tf.reduce_mean(alfa * tf.math.log(pdf) - q_min))
  </code></pre>
  <p>where &quot;pdf&quot; is the value of the probability density function of the normal distribution computed from predicted action (not clipped), mean and variance.</p>
  <p>Since the action space of LunarLanderContinuous-v2 is continuous and the possible actions are 2, my idea is to predict means μ, variances σ and a correlation ρ parameter in order to build a <strong>Multivariate normal distribution</strong> using tfp.distributions.MultivariateNormalTriL and sample actions from it.</p>
  <p>My problem is that, according to the Appendix C of the paper <a href="https://arxiv.org/pdf/1801.01290.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1801.01290.pdf</a>, I have to compute the likelihoods of the bounded actions using</p>
  <p><a href="https://i.sstatic.net/gD4xa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gD4xa.png" alt="enter image description here" /></a></p>
  <p>but I don't know how to do that because the first term is a single number due to it come from a Multivariate normal distribution, instead the second term contains 2 elements, one for each action.
  Has anyone any idea to code it?</p>
  
Tags: <reinforcement-learning><tensorflow><soft-actor-critic>
Title: How to build the actor policy of Soft-Actor-Critic after sampling from a Multivariate normal distribution?
UserDisplayName: Luc-dotcom
ThreadText: # How to build the actor policy of Soft-Actor-Critic after sampling from a Multivariate normal distribution?
  
  (Asked by: username_0 on 2022-07-29)
  
  I'm trying to solve **LunarLanderContinuous-v2** (<https://www.gymlibrary.ml/environments/box2d/lunar_lander/>) using **Soft Actor-Critic** algorithm (following the pseudocode above)
  
  ![Text](https://spinningup.openai.com/en/latest/_images/math/c01f4994ae4aacf299a6b3ceceedfe0a14d4b874.svg)
  
  To update the actor policy (step 14) I understood that I need to:
  
  1. sample an action from a normal distribution using mean and variance that are the output of the policy network
  2. squash the action with tanh to have action bounds [-1, 1]
  3. compute the log\_prob, the log probability density of the sample
  4. do the following operation in tensorflow
  
  ```
  # STEP 14
  bounded_actions_sample, predicted_next_actions_sample = policy_network.predict_actions(batch_observation)
  pdf = get_pdf(predicted_next_actions_sample, batch_observation)
  q_min = tf.math.minimum(q_network_1(batch_observation, bounded_actions_sample), q_network_2(batch_observation, bounded_actions_sample))
          
  loss_policy_network = tf.reduce_mean(alfa * tf.math.log(pdf) - q_min))
  
  ```
  
  where "pdf" is the value of the probability density function of the normal distribution computed from predicted action (not clipped), mean and variance.
  
  Since the action space of LunarLanderContinuous-v2 is continuous and the possible actions are 2, my idea is to predict means μ, variances σ and a correlation ρ parameter in order to build a **Multivariate normal distribution** using tfp.distributions.MultivariateNormalTriL and sample actions from it.
  
  My problem is that, according to the Appendix C of the paper <https://arxiv.org/pdf/1801.01290.pdf>, I have to compute the likelihoods of the bounded actions using
  
  [![enter image description here](https://i.sstatic.net/gD4xa.png)](https://i.sstatic.net/gD4xa.png)
  
  but I don't know how to do that because the first term is a single number due to it come from a Multivariate normal distribution, instead the second term contains 2 elements, one for each action.
  Has anyone any idea to code it?
  
  **Tags:** reinforcement-learning, tensorflow, soft-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-30. Score: 0 | Accepted answer
  
  I don't really understand the confusion, as both terms are scalars (where $u\_i$ are the components of $u$).
  
  So if $u\sim \mathcal{N}(\mu, \Sigma)$, then the first term would be
  
  $$\log \mu(u\mid s)=-\frac{D}{2}\log(2\pi)-\frac{1}{2}\log\vert \Sigma\vert - \frac{1}{2}(u-\mu)^T\Sigma^{-1}(u-\mu) $$
  
  However, SAC assumes that $\Sigma$ is diagonal, which makes the expression easier, that is
  
  $$\log \pi(a\mid s)=-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum\_{i=1}^D\log\sigma\_i - \frac{1}{2}\sum\_{i=1}^D\frac{(u\_i-\mu\_i)^2}{\sigma\_i}-\sum\_{i=1}^D\log(1-\tanh ^2u\_i) $$
  
  If there is a correlation between the individual actions, you would have to implement the first expression.
  
  An explicit implementation I have come across in Pytorch is the following: (where $\sqrt {2\pi} \approx2.5066$)
  
  ```
  sampled_u = self.sample_z(mua_tensor.data, siga_tensor.data).data
  sampled_a = torch.tanh(sampled_u)
  
  log_pi_exp = torch.sum(- (mua_tensor.data - sampled_u.data).pow(2)
                         / (siga_tensor.data.pow(2)) / 2
                         - torch.log(siga_tensor.data * torch.tensor(2.5066)),
                         dim=-1, keepdim=True)
  log_pi_exp -= torch.sum(torch.log(1.0 - sampled_a.pow(2) + EPS), dim=-1, keepdim=True)
  
  ```
  
  However, this is not efficient and there are undoubtedly specific functions in Tensorflow that make this implicitly, but I hope this provides more intuition.
  
  > **Comments:**
  > **username_0**: You are right, I got confused with the second term of the equation. Thanks for your explanation!  
  
  ---
  
  