Answers.Body: <p>In the world of theory, <a href="https://arxiv.org/pdf/2012.01399.pdf" rel="nofollow noreferrer">Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER</a> claims to show that PPO eventually converges to a local (but not necessarily global) optimum.</p>
  <p>In practice, good convergence is somewhat luck-based. Among other problems, the number of training steps is finite, the optimal solution might not be explored, and the model might approach the optimal solution then suffer catastrophic interference/forgetting.</p>
  <p>And it's even possible that the human gait isn't the optimal solution of that task. OpenAI has been accused of <a href="https://thegradient.pub/an-epidemic-of-ai-misinformation/" rel="nofollow noreferrer">cherry-picking or misrepresenting results for good PR</a> at times, and a human gait gives better publicity than a weird, but higher scoring, solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T13:59:59.600Z
Answers.LastActivityDate: 2022-06-15T15:25:57.823Z
Answers.LastEditDate: 2022-06-15T15:25:57.823Z
Answers.UserDisplayName: Lee Reeves
Answers.Body: <p>Even if you were reaching the global optimum for the PPO loss, it wouldn't mean that the learnt policy would resemble the human stride. Nothing in the loss enforces that.</p>
  <p>More generally in RL, the policies we use as humans to &quot;solve&quot; a task aren't necessarily the optimal ones (in the sense of reaching the highest possible reward, which is what RL methods aim to do), see for example the <a href="https://www.deepmind.com/blog/innovations-of-alphago" rel="nofollow noreferrer">previously unseen strategies which AlphaGo employs to beat humans at Go</a>.</p>
  <p>That's even more the case for environments which use an imperfect physics simulator, e.g. the humanoid here can't dislocate its member and can do very jerky movements: maybe these jerky movements achieve a higher reward than a human gait.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T20:47:24.563Z
Answers.LastActivityDate: 2022-06-16T20:47:24.563Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>I'm trying to &quot;solve&quot; the OpenAI gym environment &quot;Humanoid-v3&quot;  using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5.5k). However, the learned policies do not yet resemble the human stride (like in the <a href="https://openai.com/blog/openai-baselines-ppo/" rel="nofollow noreferrer">PPO blog post</a>), which brought up a question.</p>
  <p>Should the algorithm always converge toward the global optimum (given good hyperparameters)? Or is a good convergence somewhat luck-based and you may need multiple training processes?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-15T13:37:21.473Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-15T13:51:29.707Z
Comments.Text: Please, **ask only one question**. If you have multiple questions (even if somehow related), it's a good idea to split them into multiple posts.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><convergence><proximal-policy-optimization><gym>
Title: Should PPO always converge toward the global optimum?
UserDisplayName: pjungk
ThreadText: # Should PPO always converge toward the global optimum?
  
  (Asked by: username_0 on 2022-06-15)
  
  I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5.5k). However, the learned policies do not yet resemble the human stride (like in the [PPO blog post](https://openai.com/blog/openai-baselines-ppo/)), which brought up a question.
  
  Should the algorithm always converge toward the global optimum (given good hyperparameters)? Or is a good convergence somewhat luck-based and you may need multiple training processes?
  
  **Tags:** reinforcement-learning, convergence, proximal-policy-optimization, gym
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  > **username_2**: Please, **ask only one question**. If you have multiple questions (even if somehow related), it's a good idea to split them into multiple posts.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2022-06-15. Score: 0 | Accepted answer
  
  In the world of theory, [Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER](https://arxiv.org/pdf/2012.01399.pdf) claims to show that PPO eventually converges to a local (but not necessarily global) optimum.
  
  In practice, good convergence is somewhat luck-based. Among other problems, the number of training steps is finite, the optimal solution might not be explored, and the model might approach the optimal solution then suffer catastrophic interference/forgetting.
  
  And it's even possible that the human gait isn't the optimal solution of that task. OpenAI has been accused of [cherry-picking or misrepresenting results for good PR](https://thegradient.pub/an-epidemic-of-ai-misinformation/) at times, and a human gait gives better publicity than a weird, but higher scoring, solution.
  
  
  ---
  
  ### Answer by username_4 on 2022-06-16. Score: 0
  
  Even if you were reaching the global optimum for the PPO loss, it wouldn't mean that the learnt policy would resemble the human stride. Nothing in the loss enforces that.
  
  More generally in RL, the policies we use as humans to "solve" a task aren't necessarily the optimal ones (in the sense of reaching the highest possible reward, which is what RL methods aim to do), see for example the [previously unseen strategies which AlphaGo employs to beat humans at Go](https://www.deepmind.com/blog/innovations-of-alphago).
  
  That's even more the case for environments which use an imperfect physics simulator, e.g. the humanoid here can't dislocate its member and can do very jerky movements: maybe these jerky movements achieve a higher reward than a human gait.
  
  
  ---
  
  