Answers.Body: <p>In the standard Cartpole environment, the episode ends either at 500 timesteps or when the Cartpole falls down. Expert demonstrations show how to keep the Cartpole upright for 500 timesteps.</p>
  <p>One degenerate way for Generative Adversarial Imitation Learning or Adversarial Inverse Reinforcement Learning to train an expert policy is to assign a positive reward to every action taken by the generator policy, regardless of whether it mirrors the expert actions. With a positive reward, the RL algorithm that is training the generator policy learns to favor prolonging the episode.</p>
  <p>Hence the episode length is a form of side-channel information that makes the imitation learning task easier, and we want to exclude it by choosing environments with fixed episode length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T23:23:31.233Z
Answers.Text: Thank you for your answer @twink_ml. This puts me in a difficult situation, the enviroment I´m using in my project cannot have a fixed length, it could finish well before any imposed length.
Answers.UserDisplayName: aletelecomm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T23:52:49.313Z
Answers.Text: You can use something like https://seals.readthedocs.io/en/latest/common/util.html#seals.util.AbsorbAfterDoneWrapper
  
  to make your environment fixed horizon T, given that the horizon is upper bounded by T.
Answers.UserDisplayName: twink_ml
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T17:04:33.453Z
Answers.LastActivityDate: 2022-11-29T17:04:33.453Z
Answers.UserDisplayName: twink_ml
Body: <p>In this page <a href="https://imitation.readthedocs.io/en/latest/getting-started/variable-horizon.html" rel="nofollow noreferrer">Limitations on horizon length</a> from the <a href="https://imitation.readthedocs.io/en/latest/" rel="nofollow noreferrer">Imitation library</a>, the authors recommend that the user sticks to fixed horizon experiments because there could be &quot;information leak&quot; otherwise.</p>
  <p>I'm having problems understanding this term, how can the information leak?</p>
  <p>Can someone please explain this with an example, or something that might help me get some intuition about the issue?</p>
  
Tags: <reinforcement-learning><terminology><imitation-learning><episodes>
Title: Why could there be "information leak" if we do not use fixed horizons?
UserDisplayName: aletelecomm
ThreadText: # Why could there be "information leak" if we do not use fixed horizons?
  
  (Asked by: username_0 on 2022-11-18)
  
  In this page [Limitations on horizon length](https://imitation.readthedocs.io/en/latest/getting-started/variable-horizon.html) from the [Imitation library](https://imitation.readthedocs.io/en/latest/), the authors recommend that the user sticks to fixed horizon experiments because there could be "information leak" otherwise.
  
  I'm having problems understanding this term, how can the information leak?
  
  Can someone please explain this with an example, or something that might help me get some intuition about the issue?
  
  **Tags:** reinforcement-learning, terminology, imitation-learning, episodes
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-29. Score: 1
  
  In the standard Cartpole environment, the episode ends either at 500 timesteps or when the Cartpole falls down. Expert demonstrations show how to keep the Cartpole upright for 500 timesteps.
  
  One degenerate way for Generative Adversarial Imitation Learning or Adversarial Inverse Reinforcement Learning to train an expert policy is to assign a positive reward to every action taken by the generator policy, regardless of whether it mirrors the expert actions. With a positive reward, the RL algorithm that is training the generator policy learns to favor prolonging the episode.
  
  Hence the episode length is a form of side-channel information that makes the imitation learning task easier, and we want to exclude it by choosing environments with fixed episode length.
  
  > **Comments:**
  > **username_0**: Thank you for your answer @username_1. This puts me in a difficult situation, the enviroment I´m using in my project cannot have a fixed length, it could finish well before any imposed length.  
  > **username_1**: You can use something like https://seals.readthedocs.io/en/latest/common/util.html#seals.util.AbsorbAfterDoneWrapper  to make your environment fixed horizon T, given that the horizon is upper bounded by T.  
  
  ---
  
  