Answers.Body: <p>State 1 has zero states on the left, and 100 on the right. The normal probability of getting to a state is 1/200; the remainder from 1.0 indicates termination.</p>
  <p>So State 1: 100 x 1/200 = 1/2 = <strong>0.5</strong>, which is the probability of reaching <em>any</em> of the 100 states on the right. The remaining <strong>0.5</strong> is thus assigned to terminating, as there are no left states that can be reached.</p>
  <p>State 950: you have 100 states on the left (so a <strong>0.5</strong> probability of reaching any of them), but only 50 states on the right. So you have 50 x 1/200 = 1/4 = <strong>0.25</strong>, which is the probability of reaching any of the 50 states on the right. This leaves a probability of <strong>0.25</strong> to terminate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-31T08:14:17.447Z
Answers.LastActivityDate: 2022-03-31T08:14:17.447Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Sutton-Barto's RL book (page 203)</p>
  <blockquote>
  <p>Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a
  1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and
  144). The states are numbered from 1 to 1000, left to right, and all episodes begin near
  the center, in state 500. State transitions are from the current state to one of the 100
  neighboring states to its left, or to one of the 100 neighboring states to its right, all with
  equal probability. Of course, if the current state is near an edge, then there may be fewer
  than 100 neighbors on that side of it. In this case, all the probability that would have
  gone into those missing neighbors goes into the probability of terminating on that side
  (thus, state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance
  of terminating on the right). As usual, termination on the left produces a reward of
  −1, and termination on the right produces a reward of +1. All other transitions have a
  reward of zero. We use this task as a running example throughout this section.</p>
  </blockquote>
  <p>Question:  I did not understand how state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right. Could someone explain this calculation in detail?</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?
UserDisplayName: DSPinfinity
ThreadText: # How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?
  
  (Asked by: username_0 on 2022-03-30)
  
  Sutton-Barto's RL book (page 203)
  
  > Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a
  > 1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and
  > 144). The states are numbered from 1 to 1000, left to right, and all episodes begin near
  > the center, in state 500. State transitions are from the current state to one of the 100
  > neighboring states to its left, or to one of the 100 neighboring states to its right, all with
  > equal probability. Of course, if the current state is near an edge, then there may be fewer
  > than 100 neighbors on that side of it. In this case, all the probability that would have
  > gone into those missing neighbors goes into the probability of terminating on that side
  > (thus, state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance
  > of terminating on the right). As usual, termination on the left produces a reward of
  > −1, and termination on the right produces a reward of +1. All other transitions have a
  > reward of zero. We use this task as a running example throughout this section.
  
  Question: I did not understand how state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right. Could someone explain this calculation in detail?
  
  **Tags:** reinforcement-learning, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-31. Score: 2 | Accepted answer
  
  State 1 has zero states on the left, and 100 on the right. The normal probability of getting to a state is 1/200; the remainder from 1.0 indicates termination.
  
  So State 1: 100 x 1/200 = 1/2 = **0.5**, which is the probability of reaching *any* of the 100 states on the right. The remaining **0.5** is thus assigned to terminating, as there are no left states that can be reached.
  
  State 950: you have 100 states on the left (so a **0.5** probability of reaching any of them), but only 50 states on the right. So you have 50 x 1/200 = 1/4 = **0.25**, which is the probability of reaching any of the 50 states on the right. This leaves a probability of **0.25** to terminate.
  
  
  ---
  
  