Answers.Body: <p>Underpinning an epsilon-greedy policy is a deterministic greedy policy.  What they are saying here is that this underlying policy has been near optimal for quite a few episodes.</p>
  <p>They are still talking about this deterministic greedy policy when discussing the inset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T00:17:13.557Z
Answers.Text: I am confused by your explanation as well.  is the one given in figure (from S to G)  greedy or epsilon-greedy?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T00:25:22.070Z
Answers.Text: My interpretation is greedy as it is the optimal path in this scenario.  Theoretically it could be a trajectory from epsilon greedy under which the epsilon is sufficiently low that all moves were made deterministically or that an exploration move was taken but it happened to be the same as the greedy move.  The simpler explanation is that it is simply greedy.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-24T02:53:37.263Z
Answers.LastActivityDate: 2024-04-24T02:53:37.263Z
Answers.UserDisplayName: foreverska
Body: <p>Sutton-Barto, Chapter 6, page:130</p>
  <blockquote>
  <p>By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown
  inset); continued <span class="math-container">$\epsilon$</span>-greedy exploration kept the average episode length at about 17 steps,
  two more than the minimum of 15.
  <a href="https://i.sstatic.net/WxqpJIvw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WxqpJIvw.png" alt="enter image description here" /></a></p>
  </blockquote>
  <p>What is meant here by the first sentence: &quot;By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown inset)&quot;?</p>
  <p>I am confused also: the applied policy is <span class="math-container">$\epsilon$</span>-greedy SARSA but in the sentence there is &quot;greedy policy&quot;. Is the path shown from S to G for greedy policy, or for SARSA?</p>
  
Tags: <sutton-barto><temporal-difference-methods>
Title: Unclear sentence in Sutton-Barto in Temporal-Difference chapter
UserDisplayName: DSPinfinity
ThreadText: # Unclear sentence in Sutton-Barto in Temporal-Difference chapter
  
  (Asked by: username_0 on 2024-04-23)
  
  Sutton-Barto, Chapter 6, page:130
  
  > By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown
  > inset); continued $\epsilon$-greedy exploration kept the average episode length at about 17 steps,
  > two more than the minimum of 15.
  > [![enter image description here](https://i.sstatic.net/WxqpJIvw.png)](https://i.sstatic.net/WxqpJIvw.png)
  
  What is meant here by the first sentence: "By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown inset)"?
  
  I am confused also: the applied policy is $\epsilon$-greedy SARSA but in the sentence there is "greedy policy". Is the path shown from S to G for greedy policy, or for SARSA?
  
  **Tags:** sutton-barto, temporal-difference-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-24. Score: 2 | Accepted answer
  
  Underpinning an epsilon-greedy policy is a deterministic greedy policy. What they are saying here is that this underlying policy has been near optimal for quite a few episodes.
  
  They are still talking about this deterministic greedy policy when discussing the inset.
  
  > **Comments:**
  > **username_0**: I am confused by your explanation as well.  is the one given in figure (from S to G)  greedy or epsilon-greedy?  
  > **username_1**: My interpretation is greedy as it is the optimal path in this scenario.  Theoretically it could be a trajectory from epsilon greedy under which the epsilon is sufficiently low that all moves were made deterministically or that an exploration move was taken but it happened to be the same as the greedy move.  The simpler explanation is that it is simply greedy.  
  
  ---
  
  