Answers.Body: <p>In this case yes, <span class="math-container">$J$</span> is the big <span class="math-container">$\min$</span> expression and you apply Adam on that. But be careful because they say they do ascent, but automatic differentiation software usually minimizes given function so your <span class="math-container">$J$</span> would be <span class="math-container">$−\min(⋅)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-13T19:38:21.853Z
Answers.LastActivityDate: 2020-03-13T19:38:21.853Z
Answers.UserDisplayName: Brale
Body: <p>I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : 
  <a href="https://i.sstatic.net/9RiYK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9RiYK.png" alt="enter image description here"></a> </p>
  
  <p>This line is giving me a hard time : </p>
  
  <p><a href="https://i.sstatic.net/XjNZz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XjNZz.png" alt="enter image description here"></a></p>
  
  <p>What does the <span class="math-container">$\operatorname{argmax}$</span> mean, in this context? They are also talking about updating the policy with a gradient ascent. So, is taking argmax with respect to <span class="math-container">$\theta$</span> the same as doing:</p>
  
  <p><a href="https://i.sstatic.net/rGLzR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rGLzR.png" alt="enter image description here"></a></p>
  
  <p>where <span class="math-container">$J$</span> is the min() function?</p>
  
Tags: <reinforcement-learning><gradient-descent><policy-gradients><proximal-policy-optimization>
Title: What is the purpose of argmax in the PPO algorithm?
UserDisplayName: Samuel Beaussant
ThreadText: # What is the purpose of argmax in the PPO algorithm?
  
  (Asked by: username_0 on 2020-03-13)
  
  I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website :
  [![enter image description here](https://i.sstatic.net/9RiYK.png)](https://i.sstatic.net/9RiYK.png)
  
  This line is giving me a hard time :
  
  [![enter image description here](https://i.sstatic.net/XjNZz.png)](https://i.sstatic.net/XjNZz.png)
  
  What does the $\operatorname{argmax}$ mean, in this context? They are also talking about updating the policy with a gradient ascent. So, is taking argmax with respect to $\theta$ the same as doing:
  
  [![enter image description here](https://i.sstatic.net/rGLzR.png)](https://i.sstatic.net/rGLzR.png)
  
  where $J$ is the min() function?
  
  **Tags:** reinforcement-learning, gradient-descent, policy-gradients, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-13. Score: 1
  
  In this case yes, $J$ is the big $\min$ expression and you apply Adam on that. But be careful because they say they do ascent, but automatic differentiation software usually minimizes given function so your $J$ would be $−\min(⋅)$.
  
  
  ---
  
  