Answers.Body: <blockquote>
  <p>What does &quot;adding one additional step to the policy&quot; mean?</p>
  </blockquote>
  <p>This is an informal shorthand for how many backup steps from a non-zero reward or learned value will receive a meaningful update. That is, an update with actual data other than the initial bootstrap.</p>
  <p>This is due to three things:</p>
  <ul>
  <li><p>Bootstrap updates in basic TD only look one timestep ahead, e.g. from <span class="math-container">$s,a$</span> to <span class="math-container">$s',a'$</span>, with the update <span class="math-container">$Q(s, a) = Q(s, a) + \alpha(r + \gamma Q(s', a') - Q(s, a))$</span></p>
  </li>
  <li><p>Only the last step in the example environment has a non-zero reward.</p>
  </li>
  <li><p>Only values that have already received at least one &quot;meaningful&quot; update are themselves useful in bootstrapping real data back to earlier time steps. Without such an update, they only have initial arbitrary values.</p>
  </li>
  </ul>
  <p>With background planning, as in Dyna, there is a chance of choosing to base the update on a state, action pair that has already been updated with some non-zero* data, whilst the agent is exploring elsewhere.</p>
  <p>Without background planning, and only using single-step backups, initially only the ends of episodes will receive updates with meaningful data.</p>
  <p>You can also use multi-step backups, or eligibility traces - as in TD(<span class="math-container">$\lambda$</span>) - to increase the number of steps that get updated at once. These options each have their own pros and cons.</p>
  <hr />
  <p>* Technically it is not non-zero that matters, but &quot;different from other experience&quot;. Initially the agent experiences a bunch of states and actions that all look the same to it in terms of its estimates of expected future reward. Before an agent can learn to choose between actions it needs to experience results that are different between them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-22T21:23:00.310Z
Answers.LastActivityDate: 2022-03-23T08:00:24.730Z
Answers.LastEditDate: 2022-03-23T08:00:24.730Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Sutton &amp; Barto's RL book at page 165 for Example 8.1, they say:</p>
  <p><a href="https://i.sstatic.net/ikqVg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ikqVg.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies found by the n = 0 and n = 50 agents halfway through the second episode. Without planning (n = 0), each episode adds only one additional step to the policy, and so only one step (the last) has been learned so far. With planning, again only one step is learned during the first episode, but here during the second episode an extensive policy has been developed that by the end of the episode will reach almost back to the start state.</p>
  </blockquote>
  <p>I have the following question:</p>
  <p>Why, without planning (n = 0), each episode adds only
  one additional step to the policy? What does &quot;adding one additional step to the policy&quot; mean?</p>
  
Tags: <reinforcement-learning><sutton-barto><dyna>
Title: Without planning, why does each episode only add one additional step to the policy?
UserDisplayName: DSPinfinity
ThreadText: # Without planning, why does each episode only add one additional step to the policy?
  
  (Asked by: username_0 on 2022-03-20)
  
  In Sutton & Barto's RL book at page 165 for Example 8.1, they say:
  
  [![enter image description here](https://i.sstatic.net/ikqVg.png)](https://i.sstatic.net/ikqVg.png)
  
  > Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies found by the n = 0 and n = 50 agents halfway through the second episode. Without planning (n = 0), each episode adds only one additional step to the policy, and so only one step (the last) has been learned so far. With planning, again only one step is learned during the first episode, but here during the second episode an extensive policy has been developed that by the end of the episode will reach almost back to the start state.
  
  I have the following question:
  
  Why, without planning (n = 0), each episode adds only
  one additional step to the policy? What does "adding one additional step to the policy" mean?
  
  **Tags:** reinforcement-learning, sutton-barto, dyna
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-22. Score: 1 | Accepted answer
  
  > What does "adding one additional step to the policy" mean?
  
  This is an informal shorthand for how many backup steps from a non-zero reward or learned value will receive a meaningful update. That is, an update with actual data other than the initial bootstrap.
  
  This is due to three things:
  
  * Bootstrap updates in basic TD only look one timestep ahead, e.g. from $s,a$ to $s',a'$, with the update $Q(s, a) = Q(s, a) + \alpha(r + \gamma Q(s', a') - Q(s, a))$
  * Only the last step in the example environment has a non-zero reward.
  * Only values that have already received at least one "meaningful" update are themselves useful in bootstrapping real data back to earlier time steps. Without such an update, they only have initial arbitrary values.
  
  With background planning, as in Dyna, there is a chance of choosing to base the update on a state, action pair that has already been updated with some non-zero\* data, whilst the agent is exploring elsewhere.
  
  Without background planning, and only using single-step backups, initially only the ends of episodes will receive updates with meaningful data.
  
  You can also use multi-step backups, or eligibility traces - as in TD($\lambda$) - to increase the number of steps that get updated at once. These options each have their own pros and cons.
  
  ---
  
  \* Technically it is not non-zero that matters, but "different from other experience". Initially the agent experiences a bunch of states and actions that all look the same to it in terms of its estimates of expected future reward. Before an agent can learn to choose between actions it needs to experience results that are different between them.
  
  
  ---
  
  