Answers.Body: <p>It's a <em>continuing task</em> in that, after failure, the agent always gets a reward of <span class="math-container">$0$</span> at each time-step <em>ad infinitum</em>. </p>
  
  <p>From the book:</p>
  
  <blockquote>
    <p>we could treat pole-balancing as a continuing task, using discounting.
    In this case the reward would be -1 on each failure and zero at all
    other times. The return at each time would then be related to <span class="math-container">$-\gamma^K$</span>, where <span class="math-container">$K$</span> is the number of time steps before failure.</p>
  </blockquote>
  
  <p>(Here I have used <span class="math-container">$\gamma$</span> as the discount factor). </p>
  
  <p>Said another way, assuming the agent fails in the (K + 1)th step the reward is <span class="math-container">$0$</span> till that step, <span class="math-container">$-1$</span> for it, and then <span class="math-container">$0$</span> for eternity. </p>
  
  <p>So the return: <span class="math-container">$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + ... + \gamma^K R_{t+K+1} + ... = -\gamma^K$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-24T09:29:33.487Z
Answers.Text: The quote and the description of it do not match here. In your text, you are describing the addition of absorbing states that allow an episodic task to be treated like a continuing one mathematically. Whilst the S&B quote is about creating an actual continuing task with resets from the episodic task (i.e. there are no absorbing states, and no episodes as such). The difference is subtle, but important in the context of the question.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-25T17:14:13.640Z
Answers.LastActivityDate: 2020-04-25T18:47:53.270Z
Answers.LastEditDate: 2020-04-25T18:47:53.270Z
Answers.UserDisplayName: Bijay Gurung
Answers.Body: <p>From <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=78" rel="nofollow noreferrer">Sutton &amp; Barto's book</a> (p. 56)</p>
  <blockquote>
  <p><strong>Example 3.4: Pole-Balancing</strong> The objective in this task is to apply forces to a cart moving along a track so as to keep a pole hinged to the cart from falling over: A failure is said to occur if the pole falls past a given angle from vertical or if the cart runs off the track. The pole is reset to vertical after each failure. This task could be treated as episodic, where the natural episodes are the repeated attempts to balance the pole. The reward in this case could be <span class="math-container">$+1$</span> for every time step on which failure did not occur, so that the return at each time would be the number of steps until failure. In this case, successful balancing forever would mean a return of infinity. Alternatively, we could treat pole-balancing as a continuing task, using discounting. In this case the reward would be <span class="math-container">$-1$</span> on each failure and zero at all other times. The return at each time would then be related to <span class="math-container">$-\gamma^{K-1}$</span>, where <span class="math-container">$K$</span> is the number of time steps before failure (as well as to the times of later failures). In either case, the return is maximized by keeping the pole balanced for as long as possible.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-16T20:24:41.020Z
Answers.LastActivityDate: 2022-04-24T09:09:26.260Z
Answers.LastEditDate: 2022-04-24T09:09:26.260Z
Answers.UserDisplayName: mimoralea
Answers.Body: <p>The key is that reinforcement learning through something like, say, <a href="https://en.wikipedia.org/wiki/State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action" rel="nofollow noreferrer">SARSA</a>, works by splitting up the state space into discrete points, and then trying to learn the best action at every point. </p>
  
  <p>To do this, it tries to pick actions that maximize the <em>reward signal</em>, possibly subject to some kind of exploration policy like <a href="https://jamesmccaffrey.wordpress.com/2017/11/30/the-epsilon-greedy-algorithm/" rel="nofollow noreferrer">epsilon-greedy</a>. </p>
  
  <p>In cart-pole, two common reward signals are:</p>
  
  <ol>
  <li>Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.</li>
  <li>Receive a reward that linearly increases with the distance the pole is off the ground.</li>
  </ol>
  
  <p>In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.</p>
  
  <p>However, an <em>offline</em> algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An <a href="https://link.springer.com/article/10.1023/A:1007465907571" rel="nofollow noreferrer">online</a> algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T07:43:33.270Z
Answers.Text: I just want to know "how can cartpole environment be continuing task even though it has an end of episode"...
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T07:51:13.800Z
Answers.Text: end of episode = terminal state = falling off the poll
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T19:25:04.600Z
Answers.Text: @user3595632 As mentioned in my answer, in the continuing version of the task, the episode does not end when the pole falls. With a suitable reward function, the agent can learn to rebalance the pole after it has fallen.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T23:57:39.087Z
Answers.Text: Then what happens to pole after it fallen?....Does it keep staying fallen state?
Answers.UserDisplayName: user3595632
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-20T16:26:28.570Z
Answers.Text: The state of the poll is real-valued (usually represented as an angle). When it falls, it leaves a narrow region that is defined as "balanced" (note, there isn't usually a single unique balanced state, it's a range of angle values). There's nothing special about this from the perspective of the state of the poll: it's just got a different angle than before. The angle will continue to change with the effects of gravity, the kinetic energy of the pole, and the force exerted by the agent's motor. The agent will receive a different (worse) reward until the pole re-enters the "balanced" region.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-04T15:07:32.890Z
Answers.LastActivityDate: 2018-08-04T15:07:32.890Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>In this case in Sutton &amp; Barto, the authors are talking about removing the episode termination. They then treat the falling pole being reset to a given (originally the starting) state distrubution as a transition with a negative reward within a longer continuing problem. This is a change to the environment descripion, and it comes with some requirements, such as needing to use discounting*.</p>
  <p>This is different to the &quot;absorbing state&quot; treatment used elsewhere to put episodic and continuing tasks on the same mathematical footing.</p>
  <p>You might use this view of a problem in any environment where the goal is to maintain a steady state. It is in part motivated by the fact that even in the episodic framing of the problem, a perfect agent would never end an episode. However, converting an episodic problem to a continuing one with resets (from limited set of transitions) as part of state transition and reward scheme may be reasonable for other purposes too.</p>
  <hr />
  <p>* Or the average reward setting, although they have not covered that option at that point in the book.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-24T09:45:54.390Z
Answers.LastActivityDate: 2022-04-24T09:45:54.390Z
Answers.UserDisplayName: Neil Slater
Body: <p>In <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Introduction to Reinforcement Learning (2nd edition)</a> by Sutton and Barto, there is an example of the Pole-Balancing problem (Example 3.4).</p>
  <p>In this example, they write that this problem can be treated as an <em>episodic task</em> or <em>continuing task</em>.</p>
  <p>I think that it can only be treated as an <em>episodic task</em> because it has an end of playing, which is falling the rod.</p>
  <p>I have no idea how this can be treated as continuing task. Even in <a href="https://gym.openai.com/envs/CartPole-v0" rel="nofollow noreferrer">OpenAI Gym cartpole env</a>, there is only the episodic mode.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-04T09:36:07.007Z
Comments.Text: If you could balance the rod indefinitely then it would be continuing
Comments.UserDisplayName: Gaius
Tags: <reinforcement-learning><sutton-barto><continuous-tasks><episodic-tasks>
Title: How can the Cart Pole problem be a continuing task?
UserDisplayName: user3595632
ThreadText: # How can the Cart Pole problem be a continuing task?
  
  (Asked by: username_0 on 2018-08-04)
  
  In [Introduction to Reinforcement Learning (2nd edition)](http://incompleteideas.net/book/RLbook2020.pdf) by Sutton and Barto, there is an example of the Pole-Balancing problem (Example 3.4).
  
  In this example, they write that this problem can be treated as an *episodic task* or *continuing task*.
  
  I think that it can only be treated as an *episodic task* because it has an end of playing, which is falling the rod.
  
  I have no idea how this can be treated as continuing task. Even in [OpenAI Gym cartpole env](https://gym.openai.com/envs/CartPole-v0), there is only the episodic mode.
  
  **Tags:** reinforcement-learning, sutton-barto, continuous-tasks, episodic-tasks
  > **Comments:**
  > **username_1**: If you could balance the rod indefinitely then it would be continuing  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2020-04-25. Score: 3
  
  It's a *continuing task* in that, after failure, the agent always gets a reward of $0$ at each time-step *ad infinitum*.
  
  From the book:
  
  > we could treat pole-balancing as a continuing task, using discounting.
  > In this case the reward would be -1 on each failure and zero at all
  > other times. The return at each time would then be related to $-\gamma^K$, where $K$ is the number of time steps before failure.
  
  (Here I have used $\gamma$ as the discount factor).
  
  Said another way, assuming the agent fails in the (K + 1)th step the reward is $0$ till that step, $-1$ for it, and then $0$ for eternity.
  
  So the return: $$G\_t = R\_{t+1} + \gamma R\_{t+2} + \gamma^2 R\_{t+3} + ... + \gamma^K R\_{t+K+1} + ... = -\gamma^K$$
  
  > **Comments:**
  > **username_3**: The quote and the description of it do not match here. In your text, you are describing the addition of absorbing states that allow an episodic task to be treated like a continuing one mathematically. Whilst the S&B quote is about creating an actual continuing task with resets from the episodic task (i.e. there are no absorbing states, and no episodes as such). The difference is subtle, but important in the context of the question.  
  
  ---
  
  ### Answer by username_4 on 2018-12-16. Score: 1
  
  From [Sutton & Barto's book](http://incompleteideas.net/book/RLbook2020.pdf#page=78) (p. 56)
  
  > **Example 3.4: Pole-Balancing** The objective in this task is to apply forces to a cart moving along a track so as to keep a pole hinged to the cart from falling over: A failure is said to occur if the pole falls past a given angle from vertical or if the cart runs off the track. The pole is reset to vertical after each failure. This task could be treated as episodic, where the natural episodes are the repeated attempts to balance the pole. The reward in this case could be $+1$ for every time step on which failure did not occur, so that the return at each time would be the number of steps until failure. In this case, successful balancing forever would mean a return of infinity. Alternatively, we could treat pole-balancing as a continuing task, using discounting. In this case the reward would be $-1$ on each failure and zero at all other times. The return at each time would then be related to $-\gamma^{K-1}$, where $K$ is the number of time steps before failure (as well as to the times of later failures). In either case, the return is maximized by keeping the pole balanced for as long as possible.
  
  
  ---
  
  ### Answer by username_5 on 2018-08-04. Score: 1
  
  The key is that reinforcement learning through something like, say, [SARSA](https://en.wikipedia.org/wiki/State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action), works by splitting up the state space into discrete points, and then trying to learn the best action at every point.
  
  To do this, it tries to pick actions that maximize the *reward signal*, possibly subject to some kind of exploration policy like [epsilon-greedy](https://jamesmccaffrey.wordpress.com/2017/11/30/the-epsilon-greedy-algorithm/).
  
  In cart-pole, two common reward signals are:
  
  1. Receive 1 reward when the pole is within a small distance of the topmost position, 0 otherwise.
  2. Receive a reward that linearly increases with the distance the pole is off the ground.
  
  In both cases, an agent can continue to learn after the pole has fallen: it will just want to move the poll back up, and will try to take actions to do so.
  
  However, an *offline* algorithm wouldn't update its policy while the agent is running. This kind of algorithm wouldn't benefit from a continuous task. An [online](https://link.springer.com/article/10.1023/A:1007465907571) algorithm, on contrast, updates its policy as it goes, and has no reason to stop between episodes, except that it might become stuck in a bad state.
  
  > **Comments:**
  > **username_0**: I just want to know "how can cartpole environment be continuing task even though it has an end of episode"...  
  > **username_0**: end of episode = terminal state = falling off the poll  
  > **username_5**: @user3595632 As mentioned in my answer, in the continuing version of the task, the episode does not end when the pole falls. With a suitable reward function, the agent can learn to rebalance the pole after it has fallen.  
  > **username_0**: Then what happens to pole after it fallen?....Does it keep staying fallen state?  
  > **username_5**: The state of the poll is real-valued (usually represented as an angle). When it falls, it leaves a narrow region that is defined as "balanced" (note, there isn't usually a single unique balanced state, it's a range of angle values). There's nothing special about this from the perspective of the state of the poll: it's just got a different angle than before. The angle will continue to change with the effects of gravity, the kinetic energy of the pole, and the force exerted by the agent's motor. The agent will receive a different (worse) reward until the pole re-enters the "balanced" region.  
  
  ---
  
  ### Answer by username_3 on 2022-04-24. Score: 0
  
  In this case in Sutton & Barto, the authors are talking about removing the episode termination. They then treat the falling pole being reset to a given (originally the starting) state distrubution as a transition with a negative reward within a longer continuing problem. This is a change to the environment descripion, and it comes with some requirements, such as needing to use discounting\*.
  
  This is different to the "absorbing state" treatment used elsewhere to put episodic and continuing tasks on the same mathematical footing.
  
  You might use this view of a problem in any environment where the goal is to maintain a steady state. It is in part motivated by the fact that even in the episodic framing of the problem, a perfect agent would never end an episode. However, converting an episodic problem to a continuing one with resets (from limited set of transitions) as part of state transition and reward scheme may be reasonable for other purposes too.
  
  ---
  
  \* Or the average reward setting, although they have not covered that option at that point in the book.
  
  
  ---
  
  