Answers.Body: <p>Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by &quot;Search Control&quot; to pick a previous state-action pair to plan for.  Planning will inevitably consume some of the random numbers so this explains &quot;about 1700&quot; and I think &quot;exactly&quot; refers to across repetitions for a given choice of n.</p>
  <p>Remember, Search Control only selects from previously experienced state-action pairs.  Before the first episode ends, no reward has been seen so sampling from previous pairs does not help in finding the solution.  So, for all values of n planning steps the performance will be the same.  Once the reward has been seen, in the second episode, planning can help propagate this reward back to the agent's present position.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-10T00:48:26.703Z
Answers.LastActivityDate: 2024-05-10T00:48:26.703Z
Answers.UserDisplayName: foreverska
Body: <p>Sutton-Barto, page 164:</p>
  <p><a href="https://i.sstatic.net/kvFnQmb8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kvFnQmb8.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/gxTNm4Iz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gxTNm4Iz.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>The main part of Figure 8.2 shows average learning curves from an experiment in
  which Dyna-Q agents were applied to the maze task. The initial action values were zero,
  the step-size parameter was <span class="math-container">$\alpha$</span> = 0.1, and the exploration parameter was <span class="math-container">$\epsilon$</span> = 0.1. When selecting greedily among actions, ties were broken randomly. The agents varied in the number of planning steps, n, they performed per real step. For each n, the curves show
  the number of steps taken by the agent to reach the goal in each episode, averaged over 30
  repetitions of the experiment. In each repetition, the initial seed for the random number
  generator was held constant across algorithms. Because of this, the first episode was
  exactly the same (about 1700 steps) for all values of n, and its data are not shown in
  the figure. After the first episode, performance improved for all values of n, but much
  more rapidly for larger values. Recall that the n = 0 agent is a nonplanning agent, using
  only direct reinforcement learning (one-step tabular Q-learning).</p>
  </blockquote>
  <p>My questions:</p>
  <p>Q1- It is said that &quot; In each repetition, the initial seed for the random number generator was held constant across algorithms&quot;.</p>
  <p>Where in the Dyna-Q algorithm, random number generator used? Is it used in the <span class="math-container">$\epsilon-greedy$</span> (part b) or in the loop (f)?</p>
  <p>Q2- It is also said that &quot;Because of this, the first episode was
  exactly the same (about 1700 steps) for all values of n, and its data are not shown in
  the figure. After the first episode, performance improved for all values of n, but much
  more rapidly for larger values.&quot;</p>
  <p>Dyna-Q is used in each step of an episode and hence it is not clear to me why  first episode was exactly the same (about 1700 steps) for all values of n.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-10T13:02:51.663Z
Comments.Text: Please, do not write titles like "unclear point..." in the title. Just write your **specific question** in the title.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><sutton-barto>
Title: Unclear points in Dyna Maze example in Sutton-Barto
UserDisplayName: DSPinfinity
ThreadText: # Unclear points in Dyna Maze example in Sutton-Barto
  
  (Asked by: username_0 on 2024-05-10)
  
  Sutton-Barto, page 164:
  
  [![enter image description here](https://i.sstatic.net/kvFnQmb8.png)](https://i.sstatic.net/kvFnQmb8.png)
  
  [![enter image description here](https://i.sstatic.net/gxTNm4Iz.png)](https://i.sstatic.net/gxTNm4Iz.png)
  
  > The main part of Figure 8.2 shows average learning curves from an experiment in
  > which Dyna-Q agents were applied to the maze task. The initial action values were zero,
  > the step-size parameter was $\alpha$ = 0.1, and the exploration parameter was $\epsilon$ = 0.1. When selecting greedily among actions, ties were broken randomly. The agents varied in the number of planning steps, n, they performed per real step. For each n, the curves show
  > the number of steps taken by the agent to reach the goal in each episode, averaged over 30
  > repetitions of the experiment. In each repetition, the initial seed for the random number
  > generator was held constant across algorithms. Because of this, the first episode was
  > exactly the same (about 1700 steps) for all values of n, and its data are not shown in
  > the figure. After the first episode, performance improved for all values of n, but much
  > more rapidly for larger values. Recall that the n = 0 agent is a nonplanning agent, using
  > only direct reinforcement learning (one-step tabular Q-learning).
  
  My questions:
  
  Q1- It is said that " In each repetition, the initial seed for the random number generator was held constant across algorithms".
  
  Where in the Dyna-Q algorithm, random number generator used? Is it used in the $\epsilon-greedy$ (part b) or in the loop (f)?
  
  Q2- It is also said that "Because of this, the first episode was
  exactly the same (about 1700 steps) for all values of n, and its data are not shown in
  the figure. After the first episode, performance improved for all values of n, but much
  more rapidly for larger values."
  
  Dyna-Q is used in each step of an episode and hence it is not clear to me why first episode was exactly the same (about 1700 steps) for all values of n.
  
  **Tags:** reinforcement-learning, sutton-barto
  > **Comments:**
  > **username_1**: Please, do not write titles like "unclear point..." in the title. Just write your **specific question** in the title.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-05-10. Score: 1 | Accepted answer
  
  Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. Planning will inevitably consume some of the random numbers so this explains "about 1700" and I think "exactly" refers to across repetitions for a given choice of n.
  
  Remember, Search Control only selects from previously experienced state-action pairs. Before the first episode ends, no reward has been seen so sampling from previous pairs does not help in finding the solution. So, for all values of n planning steps the performance will be the same. Once the reward has been seen, in the second episode, planning can help propagate this reward back to the agent's present position.
  
  
  ---
  
  