Answers.Body: <p>I <em>believe</em> this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements <em>seems</em> like it will cause you to learn strange behaviors. </p>
  
  <p>Certainly, the statement is inaccurate. We need only reach page 26 to see a counterexample: The Q-learning equation (2.1) sums over <em>all</em> actions, not just the greedy ones. This also applies to the temporal difference learning method specified in Chapter 6, which is the method Figure 1.1 is discussing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-25T06:55:23.077Z
Answers.Text: I would say that the statement is inaccurate rather than incorrect. In off-policy learning with current greedy policy as target policy, exploratory actions are not backed up to cause value estimate updates (or if you prefer, the importance sampling weighting is zero), whilst greedy moves are. When I read the book, I took the statement in that sense.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-25T11:46:08.023Z
Answers.Text: @NeilSlater that's a good point. I'll update the answer.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-25T00:14:23.983Z
Answers.LastActivityDate: 2019-10-25T11:46:25.600Z
Answers.LastEditDate: 2019-10-25T11:46:25.600Z
Answers.UserDisplayName: John Doucette
Body: <p>In Chapter 1 of the book Reinformcement Learning An Introduction 2nd Edition by Richard S. Sutton and Andrew G.Barto, there is one statement "Exploratory moves do not result in any learning".</p>
  
  <p>This sentence is in Figure 1.1.</p>
  
  <blockquote>
    <p>Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken during a game; the dashed lines represent moves that we (our reinforcement learning player) considered but did not make. Our second move was an exploratory move, meaning that it was taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory moves do not result in any learning, but each of our other moves does, causing updates as suggested by the red arrows in which estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text.</p>
  </blockquote>
  
  <p>It confuses me. In my understanding, exploration should contribute to learning in almost all RL algorithms. So, why does the book state "Exploratory moves do not result in any learning" in this case?</p>
  
Tags: <machine-learning><reinforcement-learning>
Title: Why "Exploratory moves do not result in any learning"?
UserDisplayName: Morgan Cheng
ThreadText: # Why "Exploratory moves do not result in any learning"?
  
  (Asked by: username_0 on 2019-10-24)
  
  In Chapter 1 of the book Reinformcement Learning An Introduction 2nd Edition by Richard S. Sutton and Andrew G.Barto, there is one statement "Exploratory moves do not result in any learning".
  
  This sentence is in Figure 1.1.
  
  > Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken during a game; the dashed lines represent moves that we (our reinforcement learning player) considered but did not make. Our second move was an exploratory move, meaning that it was taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory moves do not result in any learning, but each of our other moves does, causing updates as suggested by the red arrows in which estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text.
  
  It confuses me. In my understanding, exploration should contribute to learning in almost all RL algorithms. So, why does the book state "Exploratory moves do not result in any learning" in this case?
  
  **Tags:** machine-learning, reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-10-25. Score: 4 | Accepted answer
  
  I *believe* this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements *seems* like it will cause you to learn strange behaviors.
  
  Certainly, the statement is inaccurate. We need only reach page 26 to see a counterexample: The Q-learning equation (2.1) sums over *all* actions, not just the greedy ones. This also applies to the temporal difference learning method specified in Chapter 6, which is the method Figure 1.1 is discussing.
  
  > **Comments:**
  > **username_2**: I would say that the statement is inaccurate rather than incorrect. In off-policy learning with current greedy policy as target policy, exploratory actions are not backed up to cause value estimate updates (or if you prefer, the importance sampling weighting is zero), whilst greedy moves are. When I read the book, I took the statement in that sense.  
  > **username_1**: @NeilSlater that's a good point. I'll update the answer.  
  
  ---
  
  