Answers.Body: <p>I looked at this not long ago.  You need to understand that the slide is referring to an <em>optimal</em> (not expected) value function &amp; optimal Action-Value function.  Let's look at his diagram.  From left to right you have state C1 = 6 for its optimal value, C2 has 8 and C3 has 10.</p>
  <p><a href="https://i.sstatic.net/kvTBM.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kvTBM.jpg" alt="enter image description here" /></a></p>
  <p>If we take C10, we can see there are 2 choices for the transition, 1) to the pub for a reward of +1 or 2) to study for +10 reward.  There are no probabilities assigned to our decision, so we will take the action that maximizes our action-value.  So, being in C3 and deterministically choosing to study gives a reward of 10.  The action is studying and the reward is 10 so the action-value is 10 + the undiscounted value of the next state.  From there you end up in the sleep state (with value 0) with no further reward.</p>
  <p>If you are in C3 and deterministically choose the action to go to the pub, you receive a reward of +1 with 3 possibilities of states/values.  Given this action value of C3 &amp; going to pub, you can apply an expectation over values of where you end up (not in the choice you make from C3).  Since you can end up in C1, C2 or C3 with their own probabilities and values, you end up with an action-value of 8.3 by choosing to go to the pub.</p>
  <p>Likewise for calculating optimal action values for C1 and C2.  You see which deterministic action gives the maximum sum of reward and next state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-18T19:34:42.053Z
Answers.LastActivityDate: 2021-09-18T19:34:42.053Z
Answers.UserDisplayName: HonestMath
Body: <p>I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 (<a href="https://www.youtube.com/watch?v=lfHX2hHRMVQ&amp;list=PLqYmG7hTraZBiG_XpjnPrSNw-1XQaM_gB&amp;ab_channel=DeepMind" rel="nofollow noreferrer">Link to Video</a>). At 1:11:00, I could not understand how he is calculating the state-value functions for C1, C2 and C3 (nodes with values 6, 8 and 10 respectively) in the student MDP example, starting from C3 and working backwards. Can someone please explain this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T18:13:49.950Z
Comments.Text: I couldn't find any states labeled with C1 C2 or C3 at the given timestamp.
Comments.UserDisplayName: tnfru
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T18:34:53.143Z
Comments.Text: Hello. It may also be a good idea to provide at least a diagram or a screenshot of it that illustrates the problem, if available.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-11T20:06:48.320Z
Comments.Text: @tnfru Sorry about the lack of clarification! C1, C2 and C3 are the three nodes in the center of the diagram! (nodes marked with values 6, 8 and 10 respectively at the mentioned timestamp!)
Comments.UserDisplayName: Gokulakrishnan CANDASSAMY
Tags: <reinforcement-learning><markov-decision-process>
Title: Calculating state-value functions in Markov Decision Process
UserDisplayName: Gokulakrishnan CANDASSAMY
ThreadText: # Calculating state-value functions in Markov Decision Process
  
  (Asked by: username_0 on 2021-09-10)
  
  I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 ([Link to Video](https://www.youtube.com/watch?v=lfHX2hHRMVQ&list=PLqYmG7hTraZBiG_XpjnPrSNw-1XQaM_gB&ab_channel=DeepMind)). At 1:11:00, I could not understand how he is calculating the state-value functions for C1, C2 and C3 (nodes with values 6, 8 and 10 respectively) in the student MDP example, starting from C3 and working backwards. Can someone please explain this?
  
  **Tags:** reinforcement-learning, markov-decision-process
  
  > **Comments:**
  > **username_1**: I couldn't find any states labeled with C1 C2 or C3 at the given timestamp.  
  > **username_2**: Hello. It may also be a good idea to provide at least a diagram or a screenshot of it that illustrates the problem, if available.  
  > **username_0**: @username_1 Sorry about the lack of clarification! C1, C2 and C3 are the three nodes in the center of the diagram! (nodes marked with values 6, 8 and 10 respectively at the mentioned timestamp!)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-18. Score: 0
  
  I looked at this not long ago. You need to understand that the slide is referring to an *optimal* (not expected) value function & optimal Action-Value function. Let's look at his diagram. From left to right you have state C1 = 6 for its optimal value, C2 has 8 and C3 has 10.
  
  [![enter image description here](https://i.sstatic.net/kvTBM.jpg)](https://i.sstatic.net/kvTBM.jpg)
  
  If we take C10, we can see there are 2 choices for the transition, 1) to the pub for a reward of +1 or 2) to study for +10 reward. There are no probabilities assigned to our decision, so we will take the action that maximizes our action-value. So, being in C3 and deterministically choosing to study gives a reward of 10. The action is studying and the reward is 10 so the action-value is 10 + the undiscounted value of the next state. From there you end up in the sleep state (with value 0) with no further reward.
  
  If you are in C3 and deterministically choose the action to go to the pub, you receive a reward of +1 with 3 possibilities of states/values. Given this action value of C3 & going to pub, you can apply an expectation over values of where you end up (not in the choice you make from C3). Since you can end up in C1, C2 or C3 with their own probabilities and values, you end up with an action-value of 8.3 by choosing to go to the pub.
  
  Likewise for calculating optimal action values for C1 and C2. You see which deterministic action gives the maximum sum of reward and next state.
  
  
  ---
  
  