Answers.Body: <p>The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the <span class="math-container">$\epsilon$</span>-greedy policy used for learning, with a specific value of <span class="math-container">$\epsilon$</span>.</p>
  <p>However, this in no way constrains how the action values can or should be used to derive any other policy. Deriving a greedy policy from the current action values will typically be an improvement (as per the policy improvement theorem), and also a reasonable gauge of how well the agent has learned the environment so far.</p>
  <p>It is fairly common approach in SARSA when using <span class="math-container">$\epsilon$</span>-greedy policies to decay <span class="math-container">$\epsilon$</span>, and for both Q-learning and SARSA to evaluate the current greedy policy at snapshots in order to track learning progress without the complication of allowing for exploratory actions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-19T11:04:52.567Z
Answers.LastActivityDate: 2024-05-19T11:04:52.567Z
Answers.UserDisplayName: Neil Slater
Body: <p>Suuton-Barto, page 176:</p>
  <blockquote>
  <p>experiment to assess the effect empirically. To isolate the e↵ect of the update distribution,
  we used entirely one-step expected tabular updates, as defined by (8.1). In the uniform
  case, we cycled through all state–action pairs, updating each in place, and in the on-policy
  case we simulated episodes, all starting in the same state, updating each state–action pair
  that occurred under the current <span class="math-container">$\epsilon$</span>-greedy policy (<span class="math-container">$\epsilon$</span>=0.1). The tasks were undiscounted
  episodic tasks, generated randomly as follows. From each of the |S| states, two actions
  were possible, each of which resulted in one of b next states, all equally likely, with a
  different random selection of b states for each state–action pair. The branching factor, b,
  was the same for all state–action pairs. In addition, on all transitions there was a 0.1
  probability of transition to the terminal state, ending the episode. The expected reward
  on each transition was selected from a Gaussian distribution with mean 0 and variance 1.</p>
  </blockquote>
  <blockquote>
  <p>At any point in the planning process
  one can stop and exhaustively compute
  <span class="math-container">$v_{\tilde{\pi}}(s_0)$</span>, the true value of the start state
  under the greedy policy, <span class="math-container">$\tilde{\pi}$</span>, given the current
  action-value function Q, as an indication
  of how well the agent would do on
  a new episode on which it acted greedily
  (all the while assuming the model is
  correct).</p>
  </blockquote>
  <p>Question: In the first paragraph, authors say the policy used for simulation is <span class="math-container">$\epsilon$</span>-greedy and in the second paragraph, they are now specifying <span class="math-container">$\tilde{\pi}$</span> to be the greedy policy corresponding to the current action-value function Q. However, does not the current action-value function Q correspond to the <span class="math-container">$\epsilon$</span>-greedy policy?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-17T15:57:08.223Z
Comments.Text: Note in the picture there are three red lines and three blue lines. Also, please don't paste screenshots full of text. Use text. you could then use pull quotes instead of referring to highlighted bits. The graph seems irrelevant to your question, but including it and referring to colours that are in it makes it appear relevant
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-18T17:09:09.267Z
Comments.Text: I meant the underlined red and blue lines in the text, not red and blue lines in the plot. I updated this in the question.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-18T22:20:36.183Z
Comments.Text: Thanks. Still needs to be text, not a screenshot please. Screenshots of text from textbooks are not searchable on the site, and less easy for screenreaders to process.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-18T23:42:47.443Z
Comments.Text: I agree. Now, question statement is in text.
Comments.UserDisplayName: DSPinfinity
Tags: <reinforcement-learning><sutton-barto>
Title: Confusing statement in Sutton-Barto on trajectory sampling
UserDisplayName: DSPinfinity
ThreadText: # Confusing statement in Sutton-Barto on trajectory sampling
  
  (Asked by: username_0 on 2024-05-17)
  
  Suuton-Barto, page 176:
  
  > experiment to assess the effect empirically. To isolate the e↵ect of the update distribution,
  > we used entirely one-step expected tabular updates, as defined by (8.1). In the uniform
  > case, we cycled through all state–action pairs, updating each in place, and in the on-policy
  > case we simulated episodes, all starting in the same state, updating each state–action pair
  > that occurred under the current $\epsilon$-greedy policy ($\epsilon$=0.1). The tasks were undiscounted
  > episodic tasks, generated randomly as follows. From each of the |S| states, two actions
  > were possible, each of which resulted in one of b next states, all equally likely, with a
  > different random selection of b states for each state–action pair. The branching factor, b,
  > was the same for all state–action pairs. In addition, on all transitions there was a 0.1
  > probability of transition to the terminal state, ending the episode. The expected reward
  > on each transition was selected from a Gaussian distribution with mean 0 and variance 1.
  
  > At any point in the planning process
  > one can stop and exhaustively compute
  > $v\_{\tilde{\pi}}(s\_0)$, the true value of the start state
  > under the greedy policy, $\tilde{\pi}$, given the current
  > action-value function Q, as an indication
  > of how well the agent would do on
  > a new episode on which it acted greedily
  > (all the while assuming the model is
  > correct).
  
  Question: In the first paragraph, authors say the policy used for simulation is $\epsilon$-greedy and in the second paragraph, they are now specifying $\tilde{\pi}$ to be the greedy policy corresponding to the current action-value function Q. However, does not the current action-value function Q correspond to the $\epsilon$-greedy policy?
  
  **Tags:** reinforcement-learning, sutton-barto
  > **Comments:**
  > **username_1**: Note in the picture there are three red lines and three blue lines. Also, please don't paste screenshots full of text. Use text. you could then use pull quotes instead of referring to highlighted bits. The graph seems irrelevant to your question, but including it and referring to colours that are in it makes it appear relevant  
  > **username_0**: I meant the underlined red and blue lines in the text, not red and blue lines in the plot. I updated this in the question.  
  > **username_1**: Thanks. Still needs to be text, not a screenshot please. Screenshots of text from textbooks are not searchable on the site, and less easy for screenreaders to process.  
  > **username_0**: I agree. Now, question statement is in text.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-19. Score: 1 | Accepted answer
  
  The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$.
  
  However, this in no way constrains how the action values can or should be used to derive any other policy. Deriving a greedy policy from the current action values will typically be an improvement (as per the policy improvement theorem), and also a reasonable gauge of how well the agent has learned the environment so far.
  
  It is fairly common approach in SARSA when using $\epsilon$-greedy policies to decay $\epsilon$, and for both Q-learning and SARSA to evaluate the current greedy policy at snapshots in order to track learning progress without the complication of allowing for exploratory actions.
  
  
  ---
  
  