Answers.Body: <p>It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.</p>
  <p>Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate</p>
  <p><span class="math-container">$$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$</span></p>
  <p>where <span class="math-container">$s'$</span> is the state we transitioned into (note that this only holds when obtaining the <em>optimal</em> Q-value, if we were using a stochastic policy then we would have to introduce expectations).</p>
  <p>Now, this assumes that we have been <em>given/obtained</em> the optimal Q-values. To obtain them, we have to use the update rule (or any other learning algorithm) that you mentioned in your question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-21T23:09:23.990Z
Answers.LastActivityDate: 2021-02-21T23:09:23.990Z
Answers.UserDisplayName: David
Body: <p>I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:</p>
  <ul>
  <li>Learning rate = 0.1</li>
  <li>Discount factor = 0.95</li>
  <li>Exploration rate = 0.1</li>
  <li>Default reward = 0</li>
  <li>The final reward (for reaching the trophy) = 1</li>
  </ul>
  <p>After 500 episodes I got the following results:</p>
  <p><a href="https://i.sstatic.net/Rw39f.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Rw39f.png" alt="enter image description here" /></a></p>
  <p>How would I compute the optimal state-action value, for example, for state <strong>2</strong>, where the agent is standing, and action <strong>south</strong>?</p>
  <p>My intuition was to use the following update rule of the <span class="math-container">$q$</span> function:</p>
  <p><span class="math-container">$$Q[s, a] = Q[s, a] + \alpha (r + \gamma  \max_{a'}Q[s', a'] — Q[s, a])$$</span></p>
  <p>But I am not sure of it. The math doesn't add up for me (when using the update rule).</p>
  <p>I am also wondering either I should use the backup diagram to find the optimal state-action q value by propagating the reward (gained from reaching the trophy) to the state in question.</p>
  <p>For reference, this is where I learned about the <a href="https://baijayantaroy.github.io/baijayantaroy.github.io/Reinforcement_Learning_Series_03_backup_diagram/" rel="nofollow noreferrer">backup diagram</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-21T21:02:24.880Z
Comments.Text: by using the update rule in the question then you are implicitly using the backup diagram. the way to calculate the optimal q-values is using the update rule in the question.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-21T21:07:55.497Z
Comments.Text: So basically if I want to compute the optimal state 2 action South (where the agent is standing) considering the old q value =0.9 (q value of the red arrow where the agent is standing) and the max  q(s',a') is the q value of the red arrow in the state beneath it ?
Comments.UserDisplayName: Rim Sleimi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-21T21:10:10.040Z
Comments.Text: the value of taking south from the agents current location is equal to the immediate reward it receives + the (discounted) q-value for the state it transitions into and action it takes under the current policy. as you're interested in the optimal policy then you want the action to be the one that maximises the q-value so yes it would be $r + \gamma Q(s', 'south')$ where $s'$ is the state directly below the agents current state.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-21T21:18:25.343Z
Comments.Text: remember though that if you would like to calculate the values 'from scratch' then using the Q-learning update rule is how you would go about it. The way I described is just the definition of a q-value.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-21T22:25:21.610Z
Comments.Text: Yeah I realized that there is a difference between the update rule and the definition of Q value. Thanks again!
Comments.UserDisplayName: Rim Sleimi
Tags: <reinforcement-learning><q-learning><value-functions><bellman-equations>
Title: How would I compute the optimal state-action value for a certain state and action?
UserDisplayName: Rim Sleimi
ThreadText: # How would I compute the optimal state-action value for a certain state and action?
  
  (Asked by: username_0 on 2021-02-21)
  
  I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:
  
  * Learning rate = 0.1
  * Discount factor = 0.95
  * Exploration rate = 0.1
  * Default reward = 0
  * The final reward (for reaching the trophy) = 1
  
  After 500 episodes I got the following results:
  
  [![enter image description here](https://i.sstatic.net/Rw39f.png)](https://i.sstatic.net/Rw39f.png)
  
  How would I compute the optimal state-action value, for example, for state **2**, where the agent is standing, and action **south**?
  
  My intuition was to use the following update rule of the $q$ function:
  
  $$Q[s, a] = Q[s, a] + \alpha (r + \gamma \max\_{a'}Q[s', a'] — Q[s, a])$$
  
  But I am not sure of it. The math doesn't add up for me (when using the update rule).
  
  I am also wondering either I should use the backup diagram to find the optimal state-action q value by propagating the reward (gained from reaching the trophy) to the state in question.
  
  For reference, this is where I learned about the [backup diagram](https://baijayantaroy.github.io/baijayantaroy.github.io/Reinforcement_Learning_Series_03_backup_diagram/).
  
  **Tags:** reinforcement-learning, q-learning, value-functions, bellman-equations
  
  > **Comments:**
  > **username_1**: by using the update rule in the question then you are implicitly using the backup diagram. the way to calculate the optimal q-values is using the update rule in the question.  
  > **username_0**: So basically if I want to compute the optimal state 2 action South (where the agent is standing) considering the old q value =0.9 (q value of the red arrow where the agent is standing) and the max  q(s',a') is the q value of the red arrow in the state beneath it ?  
  > **username_1**: the value of taking south from the agents current location is equal to the immediate reward it receives + the (discounted) q-value for the state it transitions into and action it takes under the current policy. as you're interested in the optimal policy then you want the action to be the one that maximises the q-value so yes it would be $r + \gamma Q(s', 'south')$ where $s'$ is the state directly below the agents current state.  
  > **username_1**: remember though that if you would like to calculate the values 'from scratch' then using the Q-learning update rule is how you would go about it. The way I described is just the definition of a q-value.  
  > **username_0**: Yeah I realized that there is a difference between the update rule and the definition of Q value. Thanks again!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-21. Score: 4 | Accepted answer
  
  It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
  
  Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate
  
  $$Q(s, a) = r + \gamma \max\_{a'} Q(s', a)\;;$$
  
  where $s'$ is the state we transitioned into (note that this only holds when obtaining the *optimal* Q-value, if we were using a stochastic policy then we would have to introduce expectations).
  
  Now, this assumes that we have been *given/obtained* the optimal Q-values. To obtain them, we have to use the update rule (or any other learning algorithm) that you mentioned in your question.
  
  
  ---
  
  