Answers.Body: <blockquote>
    <p>Is it just about final states? So for <span class="math-container">$s \in S$</span> when S is not final? </p>
  </blockquote>
  
  <p>You are thinking the right way, but to represent what you mean you don't need to write out "when <span class="math-container">$s$</span> is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book.</p>
  
  <p>As this is a formal exercise from the book, I don't want to write out an answer that could be cut&amp;paste for all students.</p>
  
  <p>Instead I suggest you take a look at the notations section at the beginning of the book, and find how Sutton &amp; Barto use different set labels for all states including terminal states, and all states excluding terminal states. Also, check carefully which of those sets needs to be summed over.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T16:57:10.490Z
Answers.Text: Thank you! I guess you mean $S$ and $S^+$, so $s \in S$ already means it's not final. Still I'm not sure what's the difference between continuing and episodic cases in the context of this equation.
Answers.UserDisplayName: Jakub Bielan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T16:57:32.823Z
Answers.Text: @JakubBielan: Look at the sums
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T17:06:37.070Z
Answers.Text: I think I got it :) When I'll be able to give acknowledgement for your answer, I'll remember to do it for sure
Answers.UserDisplayName: Jakub Bielan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T16:41:06.427Z
Answers.LastActivityDate: 2019-03-07T16:41:06.427Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I found myself turning in cycles for a while, so to clarify Neil Slater's answer,</p>
  
  <p>In the beginning of the book, <span class="math-container">$S$</span> means "set of non-terminal states" and <span class="math-container">$S^+$</span> means "set of all states, including the terminal ones".</p>
  
  <p><span class="math-container">$$\sum_{s^{\prime} \in S} \sum_{r \in R} p(s^{\prime}, r | s,a) = 1, \forall s \in S, a \in A(s) \tag{3.3}$$</span></p>
  
  <p>That said, in eq. 3.3 when we define that <span class="math-container">$\forall s \in S$</span>, we say that that once in a terminal state, the formula does not apply (which is obvious because no action is ever available in a terminal state by definition).</p>
  
  <p>It does not however constraint the probability in how to "get" in a terminal state, and that is the key to answer the question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T15:36:48.543Z
Answers.LastActivityDate: 2019-04-17T18:23:17.717Z
Answers.LastEditDate: 2019-04-17T18:23:17.717Z
Answers.UserDisplayName: Gigi
Body: <blockquote>
    <p>Excercise 3.5 The equastions in Section 3.1 are for the continuing
    case and need to be modified (very slightly) to apply to episodic
    tasks. Show that you know the modifications needed by giving the
    modified version of (3.3).</p>
  </blockquote>
  
  <p><span class="math-container">$\displaystyle\sum_{s^{\prime} \in S} \displaystyle\sum_{r \in R} = p(s^{\prime}, r 
  | s,a) = 1$</span> , for all <span class="math-container">$s\in S, a \in A(s)$</span>    (3.3)</p>
  
  <p>Is it just about final states? So for <span class="math-container">$s \in S$</span> when S is not final? </p>
  
Tags: <reinforcement-learning><markov-decision-process><sutton-barto><markov-chain>
Title: Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5
UserDisplayName: Jakub Bielan
ThreadText: # Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5
  
  (Asked by: username_0 on 2019-03-07)
  
  > Excercise 3.5 The equastions in Section 3.1 are for the continuing
  > case and need to be modified (very slightly) to apply to episodic
  > tasks. Show that you know the modifications needed by giving the
  > modified version of (3.3).
  
  $\displaystyle\sum\_{s^{\prime} \in S} \displaystyle\sum\_{r \in R} = p(s^{\prime}, r
  | s,a) = 1$ , for all $s\in S, a \in A(s)$ (3.3)
  
  Is it just about final states? So for $s \in S$ when S is not final?
  
  **Tags:** reinforcement-learning, markov-decision-process, sutton-barto, markov-chain
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-03-07. Score: 2 | Accepted answer
  
  > Is it just about final states? So for $s \in S$ when S is not final?
  
  You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book.
  
  As this is a formal exercise from the book, I don't want to write out an answer that could be cut&paste for all students.
  
  Instead I suggest you take a look at the notations section at the beginning of the book, and find how Sutton & Barto use different set labels for all states including terminal states, and all states excluding terminal states. Also, check carefully which of those sets needs to be summed over.
  
  > **Comments:**
  > **username_0**: Thank you! I guess you mean $S$ and $S^+$, so $s \in S$ already means it's not final. Still I'm not sure what's the difference between continuing and episodic cases in the context of this equation.  
  > **username_1**: @JakubBielan: Look at the sums  
  > **username_0**: I think I got it :) When I'll be able to give acknowledgement for your answer, I'll remember to do it for sure  
  
  ---
  
  ### Answer by username_2 on 2019-04-17. Score: 0
  
  I found myself turning in cycles for a while, so to clarify Neil Slater's answer,
  
  In the beginning of the book, $S$ means "set of non-terminal states" and $S^+$ means "set of all states, including the terminal ones".
  
  $$\sum\_{s^{\prime} \in S} \sum\_{r \in R} p(s^{\prime}, r | s,a) = 1, \forall s \in S, a \in A(s) \tag{3.3}$$
  
  That said, in eq. 3.3 when we define that $\forall s \in S$, we say that that once in a terminal state, the formula does not apply (which is obvious because no action is ever available in a terminal state by definition).
  
  It does not however constraint the probability in how to "get" in a terminal state, and that is the key to answer the question.
  
  
  ---
  
  