Answers.Body: <p>It depends on how you have modelled the probability distributions that you are sampling from when you turn the neural network output into a specific action.</p>
  <p>If your three action dimensions are sampled independently (which would be a standard way to model a policy like this), then the probabilities or probability densities multiply. Which means the log probabilities add.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-20T17:53:39.640Z
Answers.Text: I got my actions from 3 different output layers of actor-network, and each one has its own mean and std. So I must use the sum function to add log_prob and save it in the memory.
Answers.UserDisplayName: m031n
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-20T16:15:15.150Z
Answers.LastActivityDate: 2021-08-20T16:15:15.150Z
Answers.UserDisplayName: Neil Slater
Body: <p>I try to solve a navigation problem with PPO; my actions space have three-part:</p>
  <ol>
  <li>robot linear velocity that is in [-3,3] range (getting from a tanh activation func)</li>
  <li>robot linear angular that is in [-pi/6, pi/6] range (getting from a tanh activation func)</li>
  <li>robot step-time duration that is from [0.2, 0.5, 0.8] (getting from a softmax activation func)</li>
  </ol>
  <p>The problem that I face is how to calculate the ratio of probability from this separate disturbing?
  Mean or sum? or was there another way to calculate log_prob from different distributions? Something like log_prob from multivariable distribution!</p>
  
Tags: <reinforcement-learning><probability-distribution><proximal-policy-optimization>
Title: How to calculate policy probability ratio in multiple action space
UserDisplayName: m031n
ThreadText: # How to calculate policy probability ratio in multiple action space
  
  (Asked by: username_0 on 2021-08-20)
  
  I try to solve a navigation problem with PPO; my actions space have three-part:
  
  1. robot linear velocity that is in [-3,3] range (getting from a tanh activation func)
  2. robot linear angular that is in [-pi/6, pi/6] range (getting from a tanh activation func)
  3. robot step-time duration that is from [0.2, 0.5, 0.8] (getting from a softmax activation func)
  
  The problem that I face is how to calculate the ratio of probability from this separate disturbing?
  Mean or sum? or was there another way to calculate log\_prob from different distributions? Something like log\_prob from multivariable distribution!
  
  **Tags:** reinforcement-learning, probability-distribution, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-20. Score: 0 | Accepted answer
  
  It depends on how you have modelled the probability distributions that you are sampling from when you turn the neural network output into a specific action.
  
  If your three action dimensions are sampled independently (which would be a standard way to model a policy like this), then the probabilities or probability densities multiply. Which means the log probabilities add.
  
  > **Comments:**
  > **username_0**: I got my actions from 3 different output layers of actor-network, and each one has its own mean and std. So I must use the sum function to add log_prob and save it in the memory.  
  
  ---
  
  