Answers.Body: <p>Generally you can make an agent care less about rewards in the future by decreasing the discount factor <span class="math-container">$\gamma$</span>.</p>
  <blockquote>
  <p>that this may cause problems with GAE, as actions in large episodes will have much larger advantage</p>
  </blockquote>
  <p>Depending on the particular task you are solving this might not be a problem. If a longer episode gives an agent a higher expected return, then the agent should learn to prolong the episode. Or if the episode length is independent of the agent's actions, the agent should still learn to act optimally with respect to the random distribution of episode length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T14:13:58.083Z
Answers.Text: The episode length is always predetermined (there are always certain options, but these options are taken changes the cost of the outcome). I wonder if there is a way to incorporate the length of the episode, which I already know, to the training...
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T16:42:19.140Z
Answers.Text: You can always include this in your state space -- for example, you could add an integer feature for the known episode length, and an integer feature for the number of timesteps until episode completion.
Answers.UserDisplayName: twink_ml
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T11:42:51.980Z
Answers.LastActivityDate: 2022-11-29T11:42:51.980Z
Answers.UserDisplayName: twink_ml
Body: <p>I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have much larger advantages than actions in smaller episodes simply because of the cascading addition of future rewards/costs. Is there some smart way of dealing with discounted future returns in such scenarios? Thank you.</p>
  
Tags: <proximal-policy-optimization><discount-factor><episodic-tasks>
Title: PPO: dealing with variable episodic length
UserDisplayName: Antonis Karvelas
ThreadText: # PPO: dealing with variable episodic length
  
  (Asked by: username_0 on 2022-11-28)
  
  I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have much larger advantages than actions in smaller episodes simply because of the cascading addition of future rewards/costs. Is there some smart way of dealing with discounted future returns in such scenarios? Thank you.
  
  **Tags:** proximal-policy-optimization, discount-factor, episodic-tasks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-29. Score: 0
  
  Generally you can make an agent care less about rewards in the future by decreasing the discount factor $\gamma$.
  
  > that this may cause problems with GAE, as actions in large episodes will have much larger advantage
  
  Depending on the particular task you are solving this might not be a problem. If a longer episode gives an agent a higher expected return, then the agent should learn to prolong the episode. Or if the episode length is independent of the agent's actions, the agent should still learn to act optimally with respect to the random distribution of episode length.
  
  > **Comments:**
  > **username_0**: The episode length is always predetermined (there are always certain options, but these options are taken changes the cost of the outcome). I wonder if there is a way to incorporate the length of the episode, which I already know, to the training...  
  > **username_1**: You can always include this in your state space -- for example, you could add an integer feature for the known episode length, and an integer feature for the number of timesteps until episode completion.  
  
  ---
  
  