Answers.Body: <p>It's not true that finite horizon MDP can't use a discount factor. Any discount factor <span class="math-container">$\gamma \leq 1$</span> is fine for finite horizon.</p>
  <p>Whether or not cartpole is a finite or infinite horizon depends on the environment implementation. The default environment in openai gym uses <span class="math-container">$T=200$</span> as the horizon. This is customizable, and most papers I see which use cartpole as a test environment use <span class="math-container">$T=1000$</span> instead. The task is considered &quot;solved&quot; when the agent can reach <span class="math-container">$T$</span> timesteps consistently (e.g., the last 100 episodes have all reached <span class="math-container">$T$</span> timesteps).</p>
  <p>If you want to treat cartpole as an infinite horizon task, that's fine too. Clearly you need <span class="math-container">$\gamma &lt; 1$</span> here, and you can just <span class="math-container">$T$</span> to some arbitrary large number.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-29T14:52:44.420Z
Answers.LastActivityDate: 2021-05-29T14:52:44.420Z
Answers.UserDisplayName: Taw
Body: <p>I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return:</p>
  <ul>
  <li>finite horizon undiscounted return (used for episodic problems)</li>
  </ul>
  <p><span class="math-container">$G = \sum_{t=0}^T R_t$</span></p>
  <ul>
  <li>infinite horizon discounted return (used for non-episodic problems).</li>
  </ul>
  <p><span class="math-container">$G = \sum_{t=0}^\infty \gamma^t R_t$</span></p>
  <p>However, I'm confusing that &quot;Is Cartpole episodic task?&quot;. Ideally, the simulation lasts forever.  This is my final objective (prolonging the lifetime). But it still has some termination states. Should I introduce the termination state and use it with a discounted return like:</p>
  <p><span class="math-container">$G = \sum_{t=0}^T \gamma^t R_t$</span></p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><return><finite-markov-decision-process>
Title: How to formulate discounted return in cartpole?
UserDisplayName: Ngoc Bui
ThreadText: # How to formulate discounted return in cartpole?
  
  (Asked by: username_0 on 2021-05-29)
  
  I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return:
  
  * finite horizon undiscounted return (used for episodic problems)
  
  $G = \sum\_{t=0}^T R\_t$
  
  * infinite horizon discounted return (used for non-episodic problems).
  
  $G = \sum\_{t=0}^\infty \gamma^t R\_t$
  
  However, I'm confusing that "Is Cartpole episodic task?". Ideally, the simulation lasts forever. This is my final objective (prolonging the lifetime). But it still has some termination states. Should I introduce the termination state and use it with a discounted return like:
  
  $G = \sum\_{t=0}^T \gamma^t R\_t$
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, return, finite-markov-decision-process
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-29. Score: 0 | Accepted answer
  
  It's not true that finite horizon MDP can't use a discount factor. Any discount factor $\gamma \leq 1$ is fine for finite horizon.
  
  Whether or not cartpole is a finite or infinite horizon depends on the environment implementation. The default environment in openai gym uses $T=200$ as the horizon. This is customizable, and most papers I see which use cartpole as a test environment use $T=1000$ instead. The task is considered "solved" when the agent can reach $T$ timesteps consistently (e.g., the last 100 episodes have all reached $T$ timesteps).
  
  If you want to treat cartpole as an infinite horizon task, that's fine too. Clearly you need $\gamma < 1$ here, and you can just $T$ to some arbitrary large number.
  
  
  ---
  
  