Answers.Body: <p>This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning from pixels such as the Atari games, you can easily expect 1-10M steps.</p>
  <p>You can get a feeling by looking at many solved environments, for instance here: <a href="https://github.com/Rafael1s/Deep-Reinforcement-Learning-Algorithms" rel="nofollow noreferrer">https://github.com/Rafael1s/Deep-Reinforcement-Learning-Algorithms</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-20T23:28:44.823Z
Answers.LastActivityDate: 2022-06-20T23:28:44.823Z
Answers.UserDisplayName: Rexcirus
Answers.Body: <p>It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the <a href="https://docs.cleanrl.dev/" rel="nofollow noreferrer">CleanRL</a> benchmarks, there are a few of them where they use PPO: <a href="https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist" rel="nofollow noreferrer">https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T22:17:37.533Z
Answers.Text: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T19:41:50.120Z
Answers.LastActivityDate: 2022-06-23T19:41:50.120Z
Answers.UserDisplayName: kir0ul
Body: <p>This is my model average rewards as follow image.</p>
  <p>How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?</p>
  <p>And I'm using <a href="https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html" rel="nofollow noreferrer">PPO</a> to train.</p>
  <p><a href="https://i.sstatic.net/OQFOD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OQFOD.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><deep-rl><training><hyper-parameters><proximal-policy-optimization>
Title: How many training steps does it usually take to train an RL model?
UserDisplayName: huang
ThreadText: # How many training steps does it usually take to train an RL model?
  
  (Asked by: username_0 on 2022-06-20)
  
  This is my model average rewards as follow image.
  
  How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?
  
  And I'm using [PPO](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html) to train.
  
  [![enter image description here](https://i.sstatic.net/OQFOD.png)](https://i.sstatic.net/OQFOD.png)
  
  **Tags:** reinforcement-learning, deep-rl, training, hyper-parameters, proximal-policy-optimization
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-06-20. Score: 1 | Accepted answer
  
  This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning from pixels such as the Atari games, you can easily expect 1-10M steps.
  
  You can get a feeling by looking at many solved environments, for instance here: <https://github.com/Rafael1s/Deep-Reinforcement-Learning-Algorithms>
  
  
  ---
  
  ### Answer by username_2 on 2022-06-23. Score: 0
  
  It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the [CleanRL](https://docs.cleanrl.dev/) benchmarks, there are a few of them where they use PPO: <https://wandb.ai/openrlbenchmark/openrlbenchmark/reportlist>.
  
  > **Comments:**
  > **username_3**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  