Answers.Body: <p>each episode you will calculate the return, you will then update the action value or <span class="math-container">$Q(s,a)$</span>  as the average each episode. Using the blackjack example from open <a href="https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py" rel="nofollow noreferrer">AI gym</a> and using a discount factor of 1, you get the following  </p>
  
  <p>episode 1 
  [{'state': (22, 10, False), 'reward': -1, 'action': 1}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]</p>
  
  <p><span class="math-container">$Q((22, 10, False),0)=-1$</span></p>
  
  <p><span class="math-container">$Q((17, 10, False),1)=-1$</span></p>
  
  <p><span class="math-container">$Q((12, 10, False),1)=-1$</span></p>
  
  <p>episode 2 
  [{'state': (21, 10, False), 'reward': 1, 'action': 0}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]</p>
  
  <p><span class="math-container">$Q((21, 10, False),0)=1$</span></p>
  
  <p><span class="math-container">$Q((17, 10, False),1)=0$</span></p>
  
  <p><span class="math-container">$Q((12, 10, False),1)=0$</span></p>
  
  <p>For <span class="math-container">$Q((17, 10, False),1)$</span> and <span class="math-container">$Q((12, 10, False),1)$</span> is the average return 
  i.e -1 for the first episode and 1 for the second.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T23:05:28.617Z
Answers.Text: Welcome to SE:AI!
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T13:29:11.807Z
Answers.LastActivityDate: 2020-04-22T15:23:18.137Z
Answers.LastEditDate: 2020-04-22T15:23:18.137Z
Answers.UserDisplayName: Joseph Santarcangelo
Answers.Body: <p><span class="math-container">$Q(s,a)$</span> denotes the <span class="math-container">$Q-value$</span> for the state-action pair. It means the expected returns if we start from state <span class="math-container">$s$</span>, take action <span class="math-container">$a$</span>, and act according to whatever policy we are currently following.</p>
  
  <p>Suppose we are in state <span class="math-container">$s_0$</span>, take action <span class="math-container">$a_0$</span>. To compute the returns, we would need to follow our current policy from whatever state we land up after taking <span class="math-container">$a_0$</span>, till the end of the episode, and sum up the rewards (or discounted rewards) that we get along the way.</p>
  
  <p><em>Why average of returns?</em><br>
  Because we would want to do this multiple times for a state-action pair and compute the average of all such episodes.</p>
  
  <p><em>Why multiple times?</em><br>
  Generally, the environments and the transition function would have some randomness and we don't get the same reward every time.</p>
  
  <p><em>Why would you want to compute this?</em><br>
  The idea is simple. Since our goal is to maximize the average return, if we compute Q-values for all the possible actions starting from state <span class="math-container">$s_0$</span>, then we can compare between the values and decide which action is going to be most beneficial to take from state <span class="math-container">$s_0$</span>.</p>
  
  <p>Since this is a tabular approach, when they say update the Q-function, they just mean to update the Q-values. </p>
  
  <p>As an example, suppose we are in state <span class="math-container">$s_0$</span> and can take actions <span class="math-container">$a_0$</span>, <span class="math-container">$a_1$</span>, and <span class="math-container">$a_2$</span>. We first compute the Q-values for <span class="math-container">$(s_0, a_0), (s_0,a_1), (s_0, a_2)$</span> pairs, and then we would choose the action which has the maximum Q-value out of these three. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-22T19:12:16.743Z
Answers.LastActivityDate: 2020-05-22T19:12:16.743Z
Answers.UserDisplayName: pecey
Body: <p><a href="https://i.sstatic.net/pjJFA.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pjJFA.jpg" alt="Flowchart"></a></p>
  
  <p>I'm having trouble understanding the 5th step in the flowchart.</p>
  
  <p>For the 5th step, the '<em>update the Q function by taking the average of returns</em>' is confusing. </p>
  
  <p>From what I understand, the Q function is basically the state-action pair values put in a table (the Q table). To update it means to make adjustments to the state-action pair value of the individual states and their respective actions (e.g state 1 action 1, state 3 action 1, state 3 action 2, so on and so forth). </p>
  
  <p>I'm not sure what '<em>average of returns</em>' means though. Is it asking me to take the average of the returns after <span class="math-container">$x$</span> episodes?  From my understanding, returns is the sum of rewards in a full episode (So, AVG=sum of returns for x episodes/x). </p>
  
  <p>And what do I do with that '<em>average</em>'? </p>
  
  <p>I'm a little confused when they say '<em>update the Q function</em>' because the Q function consists of many parameters that must be updated (the individual state-action pair value), and I'm not sure which one they are referring to.</p>
  
  <p>What is the point of calculating the average of returns? Since the state-action pair value for a particular state and particular action will always be the same (e.g if I always take action 3 in state 4, I will always get value=2 forever)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T17:23:15.187Z
Comments.Text: Where did you get that screenshot from? What book, slides, etc., is that from?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-16T00:13:01.313Z
Comments.Text: Hands-on reinforcement learning with python.( Packt publishing)    https://www.packtpub.com/big-data-and-business-intelligence/hands-reinforcement-learning-python
Comments.UserDisplayName: BG10
Tags: <reinforcement-learning><q-learning><monte-carlo-methods>
Title: How does Monte Carlo Exploring Starts work?
UserDisplayName: BG10
ThreadText: # How does Monte Carlo Exploring Starts work?
  
  (Asked by: username_0 on 2020-04-15)
  
  [![Flowchart](https://i.sstatic.net/pjJFA.jpg)](https://i.sstatic.net/pjJFA.jpg)
  
  I'm having trouble understanding the 5th step in the flowchart.
  
  For the 5th step, the '*update the Q function by taking the average of returns*' is confusing.
  
  From what I understand, the Q function is basically the state-action pair values put in a table (the Q table). To update it means to make adjustments to the state-action pair value of the individual states and their respective actions (e.g state 1 action 1, state 3 action 1, state 3 action 2, so on and so forth).
  
  I'm not sure what '*average of returns*' means though. Is it asking me to take the average of the returns after $x$ episodes? From my understanding, returns is the sum of rewards in a full episode (So, AVG=sum of returns for x episodes/x).
  
  And what do I do with that '*average*'?
  
  I'm a little confused when they say '*update the Q function*' because the Q function consists of many parameters that must be updated (the individual state-action pair value), and I'm not sure which one they are referring to.
  
  What is the point of calculating the average of returns? Since the state-action pair value for a particular state and particular action will always be the same (e.g if I always take action 3 in state 4, I will always get value=2 forever)
  
  **Tags:** reinforcement-learning, q-learning, monte-carlo-methods
  
  > **Comments:**
  > **username_1**: Where did you get that screenshot from? What book, slides, etc., is that from?  
  > **username_0**: Hands-on reinforcement learning with python.( Packt publishing)    https://www.packtpub.com/big-data-and-business-intelligence/hands-reinforcement-learning-python  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-04-22. Score: 1
  
  each episode you will calculate the return, you will then update the action value or $Q(s,a)$ as the average each episode. Using the blackjack example from open [AI gym](https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py) and using a discount factor of 1, you get the following
  
  episode 1
  [{'state': (22, 10, False), 'reward': -1, 'action': 1}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]
  
  $Q((22, 10, False),0)=-1$
  
  $Q((17, 10, False),1)=-1$
  
  $Q((12, 10, False),1)=-1$
  
  episode 2
  [{'state': (21, 10, False), 'reward': 1, 'action': 0}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]
  
  $Q((21, 10, False),0)=1$
  
  $Q((17, 10, False),1)=0$
  
  $Q((12, 10, False),1)=0$
  
  For $Q((17, 10, False),1)$ and $Q((12, 10, False),1)$ is the average return
  i.e -1 for the first episode and 1 for the second.
  
  > **Comments:**
  > **username_3**: Welcome to SE:AI!  
  
  ---
  
  ### Answer by username_4 on 2020-05-22. Score: 0
  
  $Q(s,a)$ denotes the $Q-value$ for the state-action pair. It means the expected returns if we start from state $s$, take action $a$, and act according to whatever policy we are currently following.
  
  Suppose we are in state $s\_0$, take action $a\_0$. To compute the returns, we would need to follow our current policy from whatever state we land up after taking $a\_0$, till the end of the episode, and sum up the rewards (or discounted rewards) that we get along the way.
  
  *Why average of returns?*  
  Because we would want to do this multiple times for a state-action pair and compute the average of all such episodes.
  
  *Why multiple times?*  
  Generally, the environments and the transition function would have some randomness and we don't get the same reward every time.
  
  *Why would you want to compute this?*  
  The idea is simple. Since our goal is to maximize the average return, if we compute Q-values for all the possible actions starting from state $s\_0$, then we can compare between the values and decide which action is going to be most beneficial to take from state $s\_0$.
  
  Since this is a tabular approach, when they say update the Q-function, they just mean to update the Q-values.
  
  As an example, suppose we are in state $s\_0$ and can take actions $a\_0$, $a\_1$, and $a\_2$. We first compute the Q-values for $(s\_0, a\_0), (s\_0,a\_1), (s\_0, a\_2)$ pairs, and then we would choose the action which has the maximum Q-value out of these three.
  
  
  ---
  
  