Answers.Body: <p>I actually feel kinda dumb writing this, but the update is made if the TOTAL environment steps is a multiple of the time horizon. I did, recode my PPO algorithm, and it seems to eventually solve the lunar lander environment, albeit very slowly (taking around 650k training steps from 1700 episodes to solve), and the training is very unstable, the agent might average around 100 points and drop to an avg of -80 in the next 500 episodes and suddenly jump back to 200 points (my lr is 3e-5).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T21:38:50.523Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T10:48:55.427Z
Answers.LastActivityDate: 2022-05-10T10:48:55.427Z
Answers.UserDisplayName: Matrix001
Body: <p>I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it comes to a rest or crashes at the end. However, my time horizon is set to 20, which leads me to think the last reward is not properly accounted for (since episode lengths arent always a multiple of 20) and thus, the agent predicts bogous values for future states which causes the poor results. This begs the question: how does ppo account for the final reward? Do we just hope the algorithm stumbles upon an episode where the length is a multiple of the time horizon, or is it that the environment is programmed to wait a few steps after finishing to ensure that the final reward gets accounted for?</p>
  
Tags: <proximal-policy-optimization>
Title: How does PPO account for the last reward?
UserDisplayName: Matrix001
ThreadText: # How does PPO account for the last reward?
  
  (Asked by: username_0 on 2022-05-02)
  
  I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it comes to a rest or crashes at the end. However, my time horizon is set to 20, which leads me to think the last reward is not properly accounted for (since episode lengths arent always a multiple of 20) and thus, the agent predicts bogous values for future states which causes the poor results. This begs the question: how does ppo account for the final reward? Do we just hope the algorithm stumbles upon an episode where the length is a multiple of the time horizon, or is it that the environment is programmed to wait a few steps after finishing to ensure that the final reward gets accounted for?
  
  **Tags:** proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-05-10. Score: 0 | Accepted answer
  
  I actually feel kinda dumb writing this, but the update is made if the TOTAL environment steps is a multiple of the time horizon. I did, recode my PPO algorithm, and it seems to eventually solve the lunar lander environment, albeit very slowly (taking around 650k training steps from 1700 episodes to solve), and the training is very unstable, the agent might average around 100 points and drop to an avg of -80 in the next 500 episodes and suddenly jump back to 200 points (my lr is 3e-5).
  
  > **Comments:**
  > **username_1**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  