Answers.Body: <p>In this case, <span class="math-container">$\pi$</span> has always been an <span class="math-container">$\epsilon$</span>-greedy policy. In every iteration, this <span class="math-container">$\pi$</span> is used to generate (<span class="math-container">$\epsilon$</span>-greedily) a trajectory from which the new <span class="math-container">$Q(s, a)$</span> values are calculated. The last line in the &quot;pseudocode&quot; tells you that the policy <span class="math-container">$\pi$</span> will be a new <span class="math-container">$\epsilon$</span>-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.</p>
  <p>If the last line was <span class="math-container">$\mu \leftarrow \epsilon\text{-greedy}(Q)$</span>, it would be an off-policy method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-26T19:20:27.563Z
Answers.Text: Not sure why you're using $\mu$ here while in the original post there's $\pi$, which is also the common letter to denote a policy (that is derived from $Q$), i.e. why did you revert my edit?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-27T18:25:19.990Z
Answers.Text: @nbro Since the last line of the algorithm is $\pi\leftarrow \epsilon\text{-greedy}(Q)$, if I want to show an example for off-policy, I need to use another letter. I used $\mu$ because it is also used in David Silver's lecture slides (https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf page5). Probably not the best example. Maybe a better way to make an example is to replace the first line with "Sample $k$th episode using $\mu$". (Edit: or maybe I could use $\pi'$ instead.)
Answers.UserDisplayName: Hai Nguyen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-22T09:44:56.163Z
Answers.LastActivityDate: 2020-08-10T09:15:05.520Z
Answers.LastEditDate: 2020-08-10T09:15:05.520Z
Answers.UserDisplayName: Hai Nguyen
Body: <p>In <a href="http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/control.pdf" rel="nofollow noreferrer">slide 16</a> of his lecture 5 of the course "Reinforcement Learning", David Silver introduced <em>GLIE Monte-Carlo Control</em>.</p>
  
  <p><a href="https://i.sstatic.net/Is4qu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Is4qu.png" alt="enter image description here"></a></p>
  
  <p>But why is it an on-policy control? The sampling follows a policy <span class="math-container">$\pi$</span> while improvement follows an <span class="math-container">$\epsilon$</span>-greedy policy, so isn't it an off-policy control?</p>
  
Tags: <reinforcement-learning><control-problem><on-policy-methods><monte-carlo-methods>
Title: Why is GLIE Monte-Carlo control an on-policy control?
UserDisplayName: fish_tree
ThreadText: # Why is GLIE Monte-Carlo control an on-policy control?
  
  (Asked by: username_0 on 2018-05-22)
  
  In [slide 16](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/control.pdf) of his lecture 5 of the course "Reinforcement Learning", David Silver introduced *GLIE Monte-Carlo Control*.
  
  [![enter image description here](https://i.sstatic.net/Is4qu.png)](https://i.sstatic.net/Is4qu.png)
  
  But why is it an on-policy control? The sampling follows a policy $\pi$ while improvement follows an $\epsilon$-greedy policy, so isn't it an off-policy control?
  
  **Tags:** reinforcement-learning, control-problem, on-policy-methods, monte-carlo-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-05-22. Score: 4 | Accepted answer
  
  In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are calculated. The last line in the "pseudocode" tells you that the policy $\pi$ will be a new $\epsilon$-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.
  
  If the last line was $\mu \leftarrow \epsilon\text{-greedy}(Q)$, it would be an off-policy method.
  
  > **Comments:**
  > **username_2**: Not sure why you're using $\mu$ here while in the original post there's $\pi$, which is also the common letter to denote a policy (that is derived from $Q$), i.e. why did you revert my edit?  
  > **username_1**: @nbro Since the last line of the algorithm is $\pi\leftarrow \epsilon\text{-greedy}(Q)$, if I want to show an example for off-policy, I need to use another letter. I used $\mu$ because it is also used in David Silver's lecture slides (https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf page5). Probably not the best example. Maybe a better way to make an example is to replace the first line with "Sample $k$th episode using $\mu$". (Edit: or maybe I could use $\pi'$ instead.)  
  
  ---
  
  