Answers.Body: <p>Sorry for putting all my questions in one post.</p>
  <p>After careful searching, this is the answer to all the questions:</p>
  <ol>
  <li>The expectation notation means mean average.</li>
  <li><span class="math-container">$r(\theta)$</span> does element-wise division of new policy and old policy probability distribution.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-28T08:06:44.527Z
Answers.LastActivityDate: 2023-12-28T08:06:44.527Z
Answers.UserDisplayName: TWTom
Body: <p>I am currently working on implementing Proximal Policy Optimization (PPO) for a reinforcement learning task, and I have a few questions regarding the objective function and probability calculations.</p>
  <ol>
  <li><p>In the PPO objective function, <span class="math-container">$$L(\theta) = E\left[\min \left(r(\theta) \cdot A, \text{clip}(r(\theta), 1 - \epsilon, 1 + \epsilon) \cdot A \right)\right]$$</span> does the expectation <span class="math-container">$E$</span> do a weighted average over the batch? If so, how are these weights determined?</p>
  </li>
  <li><p>When calculating the ratio function <span class="math-container">$$r(\theta) = \frac{\pi_{\text{new}}(a|s)}{\pi_{\text{old}}(a|s)}$$</span> would <span class="math-container">$r(\theta)$</span> be a single value or a whole array with probability distribution length?</p>
  </li>
  </ol>
  <p>I appreciate any clarification on these points, as it would greatly help me in understanding the nuances of PPO and implementing it effectively.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-26T10:00:59.610Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-26T17:13:42.820Z
Comments.Text: One question per post. Split this post into multiple ones.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: In the PPO objective function, does the expectation do a weighted average over the batch?
UserDisplayName: TWTom
ThreadText: # In the PPO objective function, does the expectation do a weighted average over the batch?
  
  (Asked by: username_0 on 2023-12-26)
  
  I am currently working on implementing Proximal Policy Optimization (PPO) for a reinforcement learning task, and I have a few questions regarding the objective function and probability calculations.
  
  1. In the PPO objective function, $$L(\theta) = E\left[\min \left(r(\theta) \cdot A, \text{clip}(r(\theta), 1 - \epsilon, 1 + \epsilon) \cdot A \right)\right]$$ does the expectation $E$ do a weighted average over the batch? If so, how are these weights determined?
  2. When calculating the ratio function $$r(\theta) = \frac{\pi\_{\text{new}}(a|s)}{\pi\_{\text{old}}(a|s)}$$ would $r(\theta)$ be a single value or a whole array with probability distribution length?
  
  I appreciate any clarification on these points, as it would greatly help me in understanding the nuances of PPO and implementing it effectively.
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  > **username_2**: One question per post. Split this post into multiple ones.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-12-28. Score: 0 | Accepted answer
  
  Sorry for putting all my questions in one post.
  
  After careful searching, this is the answer to all the questions:
  
  1. The expectation notation means mean average.
  2. $r(\theta)$ does element-wise division of new policy and old policy probability distribution.
  
  
  ---
  
  