Answers.Body: <blockquote>
  <p>When we choose <span class="math-container">$A_0$</span> randomly for state <span class="math-container">$S_0$</span>, do we need to update the current <span class="math-container">$\pi(S_0)$</span> to the randomly chosen <span class="math-container">$A_0$</span> as well?</p>
  </blockquote>
  <p>No.</p>
  <p>Under a given policy <span class="math-container">$\pi$</span>, the action value of a state <span class="math-container">$Q_{\pi}(s,a)$</span> is the expected return when taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>, and <em>from that point on</em> following the policy <span class="math-container">$\pi$</span>. The action you are sampling for updates can be one that the policy would not take.</p>
  <p>In general, when you are considering a specific action value - whether to sample it or update it - you are not concerned with <em>how</em> the agent got into the state and action involved. You only care what happens afterwards.</p>
  <p>For exploring starts, you are not even concerned whether reaching any given state is possible under the policy being updated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T21:15:58.657Z
Answers.Text: @ Neil Slater,  in this code (which is the Matlab implementation, they do):   "pol_pi(polInd) = unidrnd(2)-1;      % FOR EXPLORING STARTS TAKE AN INITIAL RANDOM POLICY!!!"  The link is this: https://waxworksmath.com/Authors/N_Z/Sutton/RLAI_1st_Edition/Code/Chapter_5/mc_es_bj_Script.m
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T21:18:55.700Z
Answers.Text: @ Neil Slater, and what strange is: the  code in the above link does not work if you do not do that update. However, with update it works perfectly!
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-17T07:04:37.987Z
Answers.Text: @DSPinfinity I don't know enough Matlab to tell you why they do that, but on a quick look it doesn't seem to be an implmentation of exploring starts. Instead it simulates games from the start. That still works for the blackjack environment it seems. I wouldn't recommend you use that code to understand RL.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T21:01:53.353Z
Answers.LastActivityDate: 2022-02-16T21:01:53.353Z
Answers.UserDisplayName: Neil Slater
Body: <p>Consider the following pseudocode of
  MC control with exploring starts:</p>
  <p><a href="https://i.sstatic.net/mrVT5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mrVT5.png" alt="enter image description here" /></a></p>
  <p>When we choose <span class="math-container">$A_0$</span> randomly for state <span class="math-container">$S_0$</span>, do we need to update the current <span class="math-container">$\pi(S_0)$</span> to
  the randomly chosen <span class="math-container">$A_0$</span> as well?</p>
  
Tags: <reinforcement-learning>
Title: MC control with exploring starts: do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?
UserDisplayName: DSPinfinity
ThreadText: # MC control with exploring starts: do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?
  
  (Asked by: username_0 on 2022-02-16)
  
  Consider the following pseudocode of
  MC control with exploring starts:
  
  [![enter image description here](https://i.sstatic.net/mrVT5.png)](https://i.sstatic.net/mrVT5.png)
  
  When we choose $A\_0$ randomly for state $S\_0$, do we need to update the current $\pi(S\_0)$ to
  the randomly chosen $A\_0$ as well?
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-16. Score: 1 | Accepted answer
  
  > When we choose $A\_0$ randomly for state $S\_0$, do we need to update the current $\pi(S\_0)$ to the randomly chosen $A\_0$ as well?
  
  No.
  
  Under a given policy $\pi$, the action value of a state $Q\_{\pi}(s,a)$ is the expected return when taking action $a$ in state $s$, and *from that point on* following the policy $\pi$. The action you are sampling for updates can be one that the policy would not take.
  
  In general, when you are considering a specific action value - whether to sample it or update it - you are not concerned with *how* the agent got into the state and action involved. You only care what happens afterwards.
  
  For exploring starts, you are not even concerned whether reaching any given state is possible under the policy being updated.
  
  > **Comments:**
  > **username_0**: @ username_1 username_1,  in this code (which is the Matlab implementation, they do):   "pol_pi(polInd) = unidrnd(2)-1;      % FOR EXPLORING STARTS TAKE AN INITIAL RANDOM POLICY!!!"  The link is this: https://waxworksmath.com/Authors/N_Z/Sutton/RLAI_1st_Edition/Code/Chapter_5/mc_es_bj_Script.m  
  > **username_0**: @ username_1 username_1, and what strange is: the  code in the above link does not work if you do not do that update. However, with update it works perfectly!  
  > **username_1**: @username_0 I don't know enough Matlab to tell you why they do that, but on a quick look it doesn't seem to be an implmentation of exploring starts. Instead it simulates games from the start. That still works for the blackjack environment it seems. I wouldn't recommend you use that code to understand RL.  
  
  ---
  
  