Answers.Body: <p>The expected immediate reward functions <span class="math-container">$R(s)$</span> and <span class="math-container">$R(s,a)$</span> are convenient structures for the more fundamental <span class="math-container">$R(s,a,s')$</span> (expected reward for starting in state <span class="math-container">$s$</span>, taking action <span class="math-container">$a$</span> and arriving in state <span class="math-container">$s'$</span>) or <span class="math-container">$p(r,s'|s,a)$</span> (probability of observing immediate reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span> given starting in state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>).</p>
  <p>Which one to use depends if and how you can take advantage of the structure of the MDP's reward system.</p>
  <p>To use <span class="math-container">$R(s)$</span> you need the next expected reward to depend solely on the starting state. That isn't the case for your environment, because the reward is gained when <em>arriving</em> in state <span class="math-container">$s'$</span>.</p>
  <p>To use <span class="math-container">$R(s,a)$</span> you need the next expected reward to depend on the starting state and action. Technically this is always true, even if the environment is stochastic (and the agent might not end up in the same next state <span class="math-container">$s'$</span> each time), because you can always calculate the <em>expected</em> next reward from the fundamental features of the environment.</p>
  <p>So in your case, you should use <span class="math-container">$R(s,a)$</span> because that will define which state the agent arrives in, and enable you to figure out the immediate reward.</p>
  <p>There are other formulations of the update rule which use other building blocks. For example:</p>
  <p><span class="math-container">$$V_{t+1}(s) = \text{max}_a [\sum_{r,s'} p(r,s'|s,a)(r + \gamma \cdot V_t(s'))]$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-27T12:28:19.300Z
Answers.LastActivityDate: 2024-04-27T12:28:19.300Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am facing some confusion regarding the calculation of the values for the states in a grid world.</p>
  <p><a href="https://i.sstatic.net/8LkqdnTK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8LkqdnTK.png" alt="My Grid World Example" /></a></p>
  <p>Given that this is my grid world, where the there is a reward for going to F of +1, and no other rewards for other transitions, with the discount factor of .7, would I need to calculate the state value of F, that is, <span class="math-container">$V_{i+1}(F)$</span>?</p>
  <p>There is no indication that it is a goal state. I have calculated the state values of all other states with <span class="math-container">$V(F)$</span> being calculated and it not being calculated, where the calculations results are different. However, I am unsure which is the right calculation to use.</p>
  <p>Also, I noticed that there are 2 different equations associated with the VI update</p>
  <p><span class="math-container">$$
  V_{i+1}(s) = R(s) + \gamma \cdot \max_{a} [V_i(s')]
  $$</span></p>
  <p><span class="math-container">$$
  V_{i+1}(s) = \max_{a} [R(s, a) + \gamma \cdot V_i(s')]
  $$</span></p>
  <p>From what I understand, one is a reward associated with an action, but the other with a state. However, other than that, I don't understand when to use which.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-27T12:53:48.503Z
Comments.Text: It's not clear to me why you think you may not need to calculate $V(F)$. It's also even unclearer why you think there's no indication $F$ is the goal state, while you described it as the goal state based on the reward function. Maybe I'm misinterpreting your doubts or statements. Could you please clarify this and also change the title to be your specific question? Thanks
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process><value-iteration>
Title: Value iteration in a Grid World Example
UserDisplayName: newbieCoder
ThreadText: # Value iteration in a Grid World Example
  
  (Asked by: username_0 on 2024-04-27)
  
  I am facing some confusion regarding the calculation of the values for the states in a grid world.
  
  [![My Grid World Example](https://i.sstatic.net/8LkqdnTK.png)](https://i.sstatic.net/8LkqdnTK.png)
  
  Given that this is my grid world, where the there is a reward for going to F of +1, and no other rewards for other transitions, with the discount factor of .7, would I need to calculate the state value of F, that is, $V\_{i+1}(F)$?
  
  There is no indication that it is a goal state. I have calculated the state values of all other states with $V(F)$ being calculated and it not being calculated, where the calculations results are different. However, I am unsure which is the right calculation to use.
  
  Also, I noticed that there are 2 different equations associated with the VI update
  
  $$
  V\_{i+1}(s) = R(s) + \gamma \cdot \max\_{a} [V\_i(s')]
  $$
  
  $$
  V\_{i+1}(s) = \max\_{a} [R(s, a) + \gamma \cdot V\_i(s')]
  $$
  
  From what I understand, one is a reward associated with an action, but the other with a state. However, other than that, I don't understand when to use which.
  
  **Tags:** reinforcement-learning, markov-decision-process, value-iteration
  > **Comments:**
  > **username_1**: It's not clear to me why you think you may not need to calculate $V(F)$. It's also even unclearer why you think there's no indication $F$ is the goal state, while you described it as the goal state based on the reward function. Maybe I'm misinterpreting your doubts or statements. Could you please clarify this and also change the title to be your specific question? Thanks  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-04-27. Score: 0
  
  The expected immediate reward functions $R(s)$ and $R(s,a)$ are convenient structures for the more fundamental $R(s,a,s')$ (expected reward for starting in state $s$, taking action $a$ and arriving in state $s'$) or $p(r,s'|s,a)$ (probability of observing immediate reward $r$ and next state $s'$ given starting in state $s$ and taking action $a$).
  
  Which one to use depends if and how you can take advantage of the structure of the MDP's reward system.
  
  To use $R(s)$ you need the next expected reward to depend solely on the starting state. That isn't the case for your environment, because the reward is gained when *arriving* in state $s'$.
  
  To use $R(s,a)$ you need the next expected reward to depend on the starting state and action. Technically this is always true, even if the environment is stochastic (and the agent might not end up in the same next state $s'$ each time), because you can always calculate the *expected* next reward from the fundamental features of the environment.
  
  So in your case, you should use $R(s,a)$ because that will define which state the agent arrives in, and enable you to figure out the immediate reward.
  
  There are other formulations of the update rule which use other building blocks. For example:
  
  $$V\_{t+1}(s) = \text{max}\_a [\sum\_{r,s'} p(r,s'|s,a)(r + \gamma \cdot V\_t(s'))]$$
  
  
  ---
  
  