Body: <p>I'm trying to implement the Proximal Policy Optimization (PPO) algorithm (<a href="https://github.com/ChintanTrivedi/rl-bot-football/blob/master/train.py" rel="nofollow noreferrer">code here</a>), but I am confused about certain concepts.</p>
  
  <ol>
  <li><p>What is the correct way to implement log probability of a policy (denoted by <span class="math-container">$\pi_\theta$</span> below)?
  <span class="math-container">$$
  L^{C P I}(\theta)=\hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}\left(a_{t} | s_{t}\right)}{\pi_{\theta_{\text {old }}}\left(a_{t} | s_{t}\right)} \hat{A}_{t}\right]=\hat{\mathbb{E}}_{t}\left[r_{t}(\theta) \hat{A}_{t}\right]
  $$</span>
  Let's say my old network policy output is <code>oldpolicy_probs=[0.1,0.2,0.6,0.1]</code> and new network policy output is <code>newpolicy_probs=[0.2,0.2,0.4,0.2]</code>.</p>
  
  <p>Do I take the log of this directly, or should I first multiply these with the true label <code>y_true = [0,0,1,0]</code> as implemented <a href="https://github.com/LuEE-C/PPO-Keras/blob/3bd92268d45ac4f0709f926f24392ed99252daac/Main.py#L45" rel="nofollow noreferrer">here</a>?</p></li>
  <li><p><code>ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)</code></p>
  
  <p>Once I have the ratio and I multiply it with an advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate <span class="math-container">$\hat{\mathbb{E}_t}$</span> but conceptually I don't understand what this gives us. Is my implementation above correct?</p></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-27T18:19:44.533Z
Comments.Text: I have the same question. In this scenario, you have discrete actions right? How were you able to handle this?
Comments.UserDisplayName: Ahmed Alagha
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-01T10:06:07.570Z
Comments.Text: @AhmedAlagha Please see the implementation of ppo loss [here on github](https://github.com/ChintanTrivedi/rl-bot-football/blob/master/train.py#L63).
Comments.UserDisplayName: Chintan Trivedi
Tags: <python><objective-functions><implementation><proximal-policy-optimization>
Title: Understanding log probabilities of actions in the PPO objective
UserDisplayName: Chintan Trivedi
ThreadText: # Understanding log probabilities of actions in the PPO objective
  
  (Asked by: username_0 on 2019-07-25)
  
  I'm trying to implement the Proximal Policy Optimization (PPO) algorithm ([code here](https://github.com/ChintanTrivedi/rl-bot-football/blob/master/train.py)), but I am confused about certain concepts.
  
  1. What is the correct way to implement log probability of a policy (denoted by $\pi\_\theta$ below)?
     $$
     L^{C P I}(\theta)=\hat{\mathbb{E}}\_{t}\left[\frac{\pi\_{\theta}\left(a\_{t} | s\_{t}\right)}{\pi\_{\theta\_{\text {old }}}\left(a\_{t} | s\_{t}\right)} \hat{A}\_{t}\right]=\hat{\mathbb{E}}\_{t}\left[r\_{t}(\theta) \hat{A}\_{t}\right]
     $$
     Let's say my old network policy output is `oldpolicy_probs=[0.1,0.2,0.6,0.1]` and new network policy output is `newpolicy_probs=[0.2,0.2,0.4,0.2]`.
  
     Do I take the log of this directly, or should I first multiply these with the true label `y_true = [0,0,1,0]` as implemented [here](https://github.com/LuEE-C/PPO-Keras/blob/3bd92268d45ac4f0709f926f24392ed99252daac/Main.py#L45)?
  2. `ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)`
  
     Once I have the ratio and I multiply it with an advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate $\hat{\mathbb{E}\_t}$ but conceptually I don't understand what this gives us. Is my implementation above correct?
  
  **Tags:** python, objective-functions, implementation, proximal-policy-optimization
  > **Comments:**
  > **username_1**: I have the same question. In this scenario, you have discrete actions right? How were you able to handle this?  
  > **username_0**: @AhmedAlagha Please see the implementation of ppo loss [here on github](https://github.com/ChintanTrivedi/rl-bot-football/blob/master/train.py#L63).  
  
  ---
  
  ## 0 answers
  
  