<p>each episode you will calculate the return, you will then update the action value or <span class="math-container">$Q(s,a)$</span>  as the average each episode. Using the blackjack example from open <a href="https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py" rel="nofollow noreferrer">AI gym</a> and using a discount factor of 1, you get the following  </p>

<p>episode 1 
[{'state': (22, 10, False), 'reward': -1, 'action': 1}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]</p>

<p><span class="math-container">$Q((22, 10, False),0)=-1$</span></p>

<p><span class="math-container">$Q((17, 10, False),1)=-1$</span></p>

<p><span class="math-container">$Q((12, 10, False),1)=-1$</span></p>

<p>episode 2 
[{'state': (21, 10, False), 'reward': 1, 'action': 0}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]</p>

<p><span class="math-container">$Q((21, 10, False),0)=1$</span></p>

<p><span class="math-container">$Q((17, 10, False),1)=0$</span></p>

<p><span class="math-container">$Q((12, 10, False),1)=0$</span></p>

<p>For <span class="math-container">$Q((17, 10, False),1)$</span> and <span class="math-container">$Q((12, 10, False),1)$</span> is the average return 
i.e -1 for the first episode and 1 for the second.</p>

CC BY-SA 4.0
2020-04-23T23:05:28.617Z
Welcome to SE:AI!
DukeZhou
CC BY-SA 4.0
2020-04-22T13:29:11.807Z
2020-04-22T15:23:18.137Z
2020-04-22T15:23:18.137Z
Joseph Santarcangelo
<p><span class="math-container">$Q(s,a)$</span> denotes the <span class="math-container">$Q-value$</span> for the state-action pair. It means the expected returns if we start from state <span class="math-container">$s$</span>, take action <span class="math-container">$a$</span>, and act according to whatever policy we are currently following.</p>

<p>Suppose we are in state <span class="math-container">$s_0$</span>, take action <span class="math-container">$a_0$</span>. To compute the returns, we would need to follow our current policy from whatever state we land up after taking <span class="math-container">$a_0$</span>, till the end of the episode, and sum up the rewards (or discounted rewards) that we get along the way.</p>

<p><em>Why average of returns?</em><br>
Because we would want to do this multiple times for a state-action pair and compute the average of all such episodes.</p>

<p><em>Why multiple times?</em><br>
Generally, the environments and the transition function would have some randomness and we don't get the same reward every time.</p>

<p><em>Why would you want to compute this?</em><br>
The idea is simple. Since our goal is to maximize the average return, if we compute Q-values for all the possible actions starting from state <span class="math-container">$s_0$</span>, then we can compare between the values and decide which action is going to be most beneficial to take from state <span class="math-container">$s_0$</span>.</p>

<p>Since this is a tabular approach, when they say update the Q-function, they just mean to update the Q-values. </p>

<p>As an example, suppose we are in state <span class="math-container">$s_0$</span> and can take actions <span class="math-container">$a_0$</span>, <span class="math-container">$a_1$</span>, and <span class="math-container">$a_2$</span>. We first compute the Q-values for <span class="math-container">$(s_0, a_0), (s_0,a_1), (s_0, a_2)$</span> pairs, and then we would choose the action which has the maximum Q-value out of these three. </p>

CC BY-SA 4.0
2020-05-22T19:12:16.743Z
2020-05-22T19:12:16.743Z
pecey
<p><a href="https://i.sstatic.net/pjJFA.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pjJFA.jpg" alt="Flowchart"></a></p>

<p>I'm having trouble understanding the 5th step in the flowchart.</p>

<p>For the 5th step, the '<em>update the Q function by taking the average of returns</em>' is confusing. </p>

<p>From what I understand, the Q function is basically the state-action pair values put in a table (the Q table). To update it means to make adjustments to the state-action pair value of the individual states and their respective actions (e.g state 1 action 1, state 3 action 1, state 3 action 2, so on and so forth). </p>

<p>I'm not sure what '<em>average of returns</em>' means though. Is it asking me to take the average of the returns after <span class="math-container">$x$</span> episodes?  From my understanding, returns is the sum of rewards in a full episode (So, AVG=sum of returns for x episodes/x). </p>

<p>And what do I do with that '<em>average</em>'? </p>

<p>I'm a little confused when they say '<em>update the Q function</em>' because the Q function consists of many parameters that must be updated (the individual state-action pair value), and I'm not sure which one they are referring to.</p>

<p>What is the point of calculating the average of returns? Since the state-action pair value for a particular state and particular action will always be the same (e.g if I always take action 3 in state 4, I will always get value=2 forever)</p>

CC BY-SA 4.0
2020-04-15T17:23:15.187Z
Where did you get that screenshot from? What book, slides, etc., is that from?
nbro
CC BY-SA 4.0
2020-04-16T00:13:01.313Z
Hands-on reinforcement learning with python.( Packt publishing)    https://www.packtpub.com/big-data-and-business-intelligence/hands-reinforcement-learning-python
BG10
<reinforcement-learning><q-learning><monte-carlo-methods>
How does Monte Carlo Exploring Starts work?
BG10
# How does Monte Carlo Exploring Starts work?

(Asked by: username_0 on 2020-04-15)

[![Flowchart](https://i.sstatic.net/pjJFA.jpg)](https://i.sstatic.net/pjJFA.jpg)

I'm having trouble understanding the 5th step in the flowchart.

For the 5th step, the '*update the Q function by taking the average of returns*' is confusing.

From what I understand, the Q function is basically the state-action pair values put in a table (the Q table). To update it means to make adjustments to the state-action pair value of the individual states and their respective actions (e.g state 1 action 1, state 3 action 1, state 3 action 2, so on and so forth).

I'm not sure what '*average of returns*' means though. Is it asking me to take the average of the returns after $x$ episodes? From my understanding, returns is the sum of rewards in a full episode (So, AVG=sum of returns for x episodes/x).

And what do I do with that '*average*'?

I'm a little confused when they say '*update the Q function*' because the Q function consists of many parameters that must be updated (the individual state-action pair value), and I'm not sure which one they are referring to.

What is the point of calculating the average of returns? Since the state-action pair value for a particular state and particular action will always be the same (e.g if I always take action 3 in state 4, I will always get value=2 forever)

**Tags:** reinforcement-learning, q-learning, monte-carlo-methods

> **Comments:**
> **username_1**: Where did you get that screenshot from? What book, slides, etc., is that from?  
> **username_0**: Hands-on reinforcement learning with python.( Packt publishing)    https://www.packtpub.com/big-data-and-business-intelligence/hands-reinforcement-learning-python  

---

## 2 answers

### Answer by username_2 on 2020-04-22. Score: 1

each episode you will calculate the return, you will then update the action value or $Q(s,a)$ as the average each episode. Using the blackjack example from open [AI gym](https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py) and using a discount factor of 1, you get the following

episode 1
[{'state': (22, 10, False), 'reward': -1, 'action': 1}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]

$Q((22, 10, False),0)=-1$

$Q((17, 10, False),1)=-1$

$Q((12, 10, False),1)=-1$

episode 2
[{'state': (21, 10, False), 'reward': 1, 'action': 0}, {'state': (17, 10, False), 'reward': 0, 'action': 1}, {'state': (12, 10, False), 'reward': 0.0, 'action': 1}]

$Q((21, 10, False),0)=1$

$Q((17, 10, False),1)=0$

$Q((12, 10, False),1)=0$

For $Q((17, 10, False),1)$ and $Q((12, 10, False),1)$ is the average return
i.e -1 for the first episode and 1 for the second.

> **Comments:**
> **username_3**: Welcome to SE:AI!  

---

### Answer by username_4 on 2020-05-22. Score: 0

$Q(s,a)$ denotes the $Q-value$ for the state-action pair. It means the expected returns if we start from state $s$, take action $a$, and act according to whatever policy we are currently following.

Suppose we are in state $s\_0$, take action $a\_0$. To compute the returns, we would need to follow our current policy from whatever state we land up after taking $a\_0$, till the end of the episode, and sum up the rewards (or discounted rewards) that we get along the way.

*Why average of returns?*  
Because we would want to do this multiple times for a state-action pair and compute the average of all such episodes.

*Why multiple times?*  
Generally, the environments and the transition function would have some randomness and we don't get the same reward every time.

*Why would you want to compute this?*  
The idea is simple. Since our goal is to maximize the average return, if we compute Q-values for all the possible actions starting from state $s\_0$, then we can compare between the values and decide which action is going to be most beneficial to take from state $s\_0$.

Since this is a tabular approach, when they say update the Q-function, they just mean to update the Q-values.

As an example, suppose we are in state $s\_0$ and can take actions $a\_0$, $a\_1$, and $a\_2$. We first compute the Q-values for $(s\_0, a\_0), (s\_0,a\_1), (s\_0, a\_2)$ pairs, and then we would choose the action which has the maximum Q-value out of these three.


---

