Answers.Body: <p>In the book, the phrase &quot;generate the data&quot; refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.</p>
  <p>In both the SARSA and Q learning pseudocode from the book, there is a <em>behaviour</em> policy that selects the next action to take. Other than the initial start state, this policy drives the observations that the learning process must handle - every action is directly selected by it, and due to how MDPs work, every next state and reward are indirectly influenced by it. It is this <em>behaviour</em> policy therefore that must &quot;generate the data&quot;.</p>
  <p>You can see from the pseudocode that both algorithms describe this behaviour policy as &quot;policy derived from Q (e.g. <span class="math-container">$\epsilon$</span>-greedy)&quot;.</p>
  <p>The difference in the two algorithms is in which <em>target</em> polcy is being used:</p>
  <ul>
  <li><p>In SARSA, the update is based on the Bellman equation for the same policy that generated the most recent set of data for S, A, R, S' and A' - all of these values will be directly or indirectly caused by the behaviour policy. So the whole equation uses the same data that was generated. Another way to put that is behaviour policy and target policy are the same.</p>
  </li>
  <li><p>In Q learning, the <span class="math-container">$\text{max}_a Q(S',a)$</span> from the Bellman optimality equation removes the need to use A' in the update. Effectively it is a local search for actions to find one that is potentially better than A', and runs the update as if that one was the one that was taken. This revision of A' is what makes the off-learning evaluate a different target policy to the behaviour policy that generated the rest of the data used in the update.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-12T17:20:47.823Z
Answers.LastActivityDate: 2020-07-12T17:20:47.823Z
Answers.UserDisplayName: Neil Slater
Body: <p>From the book:</p>
  <p>Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. &quot;</p>
  <p>following is stated:</p>
  <p>&quot;On-policy methods attempt to evaluate or improve the policy that is used to make decisions, whereas off-policy methods evaluate or improve a policy different from that used to generate the data.&quot;</p>
  <p>Looking at off policy:</p>
  <p><a href="https://i.sstatic.net/d9AI0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/d9AI0.png" alt="enter image description here" /></a></p>
  <p>and on-policy:</p>
  <p><a href="https://i.sstatic.net/f6liK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/f6liK.png" alt="enter image description here" /></a></p>
  <p>What is meant by &quot;generate the data&quot;? I'm confused as to what 'data' means in this context.</p>
  <p>Does &quot;generate the data&quot; translate to the actions generated by the policy ? or Does &quot;generate the data&quot; translate to the Q data state action mappings?</p>
  
Tags: <reinforcement-learning><policies><off-policy-methods>
Title: What is meant by "generate the data" in describing the difference between on-policy and off-policy?
UserDisplayName: blue-sky
ThreadText: # What is meant by "generate the data" in describing the difference between on-policy and off-policy?
  
  (Asked by: username_0 on 2020-07-12)
  
  From the book:
  
  Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. "
  
  following is stated:
  
  "On-policy methods attempt to evaluate or improve the policy that is used to make decisions, whereas off-policy methods evaluate or improve a policy different from that used to generate the data."
  
  Looking at off policy:
  
  [![enter image description here](https://i.sstatic.net/d9AI0.png)](https://i.sstatic.net/d9AI0.png)
  
  and on-policy:
  
  [![enter image description here](https://i.sstatic.net/f6liK.png)](https://i.sstatic.net/f6liK.png)
  
  What is meant by "generate the data"? I'm confused as to what 'data' means in this context.
  
  Does "generate the data" translate to the actions generated by the policy ? or Does "generate the data" translate to the Q data state action mappings?
  
  **Tags:** reinforcement-learning, policies, off-policy-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-12. Score: 3 | Accepted answer
  
  In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.
  
  In both the SARSA and Q learning pseudocode from the book, there is a *behaviour* policy that selects the next action to take. Other than the initial start state, this policy drives the observations that the learning process must handle - every action is directly selected by it, and due to how MDPs work, every next state and reward are indirectly influenced by it. It is this *behaviour* policy therefore that must "generate the data".
  
  You can see from the pseudocode that both algorithms describe this behaviour policy as "policy derived from Q (e.g. $\epsilon$-greedy)".
  
  The difference in the two algorithms is in which *target* polcy is being used:
  
  * In SARSA, the update is based on the Bellman equation for the same policy that generated the most recent set of data for S, A, R, S' and A' - all of these values will be directly or indirectly caused by the behaviour policy. So the whole equation uses the same data that was generated. Another way to put that is behaviour policy and target policy are the same.
  * In Q learning, the $\text{max}\_a Q(S',a)$ from the Bellman optimality equation removes the need to use A' in the update. Effectively it is a local search for actions to find one that is potentially better than A', and runs the update as if that one was the one that was taken. This revision of A' is what makes the off-learning evaluate a different target policy to the behaviour policy that generated the rest of the data used in the update.
  
  
  ---
  
  