Answers.Body: <p>If you consider every action independent from the others, I assume the equation might be right. Consider the case with two actions. <span class="math-container">$\pi(a_1,a_2)$</span> is the joint probability of the two actions which becomes <span class="math-container">$\pi(a_1)\cdot \pi(a_2)$</span> in case <span class="math-container">$a_1$</span> and <span class="math-container">$a_2$</span> are independent. Then, <span class="math-container">$\log(\pi(a_1,a_2))$</span> will become <span class="math-container">$\log(\pi(a_1))+\log(\pi(a_2))$</span>. I suppose this logic can be applied in case there are multiple actions.
  However, this should be considered that <span class="math-container">$\pi(a_i)$</span> should be calculated with its own mean and variance. so I assume the policy network will have 2n outputs if we have n actions. because each action requires it's own mean and variance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-20T07:33:48.260Z
Answers.LastActivityDate: 2021-06-24T13:27:12.563Z
Answers.LastEditDate: 2021-06-24T13:27:12.563Z
Answers.UserDisplayName: navid mohamadi
Body: <p>My question concerns a side question <s>(which was not answered)</s> asked here:
  <a href="https://ai.stackexchange.com/questions/4085/policy-gradients-for-multiple-continuous-actions">How can policy gradients be applied in the case of multiple continuous actions?</a></p>
  <p>I am trying to implement a simple policy gradient algorithm for a discrete multi-action reinforcement learning task. To be more precise, there are three actuators. At every time step, each of the actuators can perform one of three possible actions.</p>
  <p>Is it possible to adjust the loss function from the single action case per time step</p>
  <p><span class="math-container">$$L = \log(P(a_1)) A$$</span></p>
  <p>to the n-action case per time step like so?</p>
  <p><span class="math-container">$$L = (\log(P(a_1)) + \log(P(a_2))+ \dots + \log(P(a_n))) A$$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-26T16:02:31.177Z
Comments.Text: Welcome to AI!  Side question becoming a question in it's own right is perfectly acceptable.  Thanks for raising it.
Comments.UserDisplayName: DukeZhou
Tags: <reinforcement-learning><objective-functions><discrete-action-spaces><multi-action-rl>
Title: Extend the loss function from the single action to the n-action case per time step
UserDisplayName: Andreas Serov
ThreadText: # Extend the loss function from the single action to the n-action case per time step
  
  (Asked by: username_0 on 2018-04-26)
  
  My question concerns a side question ~~(which was not answered)~~ asked here:
  [How can policy gradients be applied in the case of multiple continuous actions?](https://ai.stackexchange.com/questions/4085/policy-gradients-for-multiple-continuous-actions)
  
  I am trying to implement a simple policy gradient algorithm for a discrete multi-action reinforcement learning task. To be more precise, there are three actuators. At every time step, each of the actuators can perform one of three possible actions.
  
  Is it possible to adjust the loss function from the single action case per time step
  
  $$L = \log(P(a\_1)) A$$
  
  to the n-action case per time step like so?
  
  $$L = (\log(P(a\_1)) + \log(P(a\_2))+ \dots + \log(P(a\_n))) A$$?
  
  **Tags:** reinforcement-learning, objective-functions, discrete-action-spaces, multi-action-rl
  
  > **Comments:**
  > **username_1**: Welcome to AI!  Side question becoming a question in it's own right is perfectly acceptable.  Thanks for raising it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-06-20. Score: 0
  
  If you consider every action independent from the others, I assume the equation might be right. Consider the case with two actions. $\pi(a\_1,a\_2)$ is the joint probability of the two actions which becomes $\pi(a\_1)\cdot \pi(a\_2)$ in case $a\_1$ and $a\_2$ are independent. Then, $\log(\pi(a\_1,a\_2))$ will become $\log(\pi(a\_1))+\log(\pi(a\_2))$. I suppose this logic can be applied in case there are multiple actions.
  However, this should be considered that $\pi(a\_i)$ should be calculated with its own mean and variance. so I assume the policy network will have 2n outputs if we have n actions. because each action requires it's own mean and variance.
  
  
  ---
  
  