Answers.Body: <blockquote>
  <p>Should the * be initialized as 0 and then calculated using the equation?</p>
  </blockquote>
  <p>This would work if you repeated the calculation multiple times, and that would be similar to a solution method called value iteration. However, it is not what is intended here. Instead, if you expand the Bellman equation for <span class="math-container">$v_{\pi}(s)$</span> using your knowledge of existing values and the policy, you will get terms in your unknown on both sides (because one action in each case will stay in the same state) which you can resolve to directly find the value for the marked squares.</p>
  <p>To work through this for the <code>*</code> on the top line, we can call the value <span class="math-container">$x$</span> and expand the Bellman equation as follows (working clockwise from &quot;move up&quot; action):</p>
  <p><span class="math-container">$$x = 0.25 * (-1 + x) + 0.25 * (-1 - 20) + 0.25 * (-1 - 18 ) + 0.25 * (-1 + 0)$$</span></p>
  <p><span class="math-container">$$3x = -1 - 21 - 19 -1$$</span></p>
  <p><span class="math-container">$$3x = -42$$</span></p>
  <p><span class="math-container">$$x = -14$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T14:07:32.547Z
Answers.LastActivityDate: 2023-09-12T14:07:32.547Z
Answers.UserDisplayName: Neil Slater
Body: <p>Consider a 4x4 grid world problem where the goal is to reach either the top left or bottom right corner. The agent can choose from four actions up,down,left,right which deterministically cause the corresponding state transitions, except that actions that would take the agent off the grid leave the state unchanged. We model this as an undiscounted,episodic task where the reward is -1 for all transitions. Suppose that the agent follows equi-probable random policy. Calculate the state values where * is placed.(Hint: Bellman expectation equation)
  <a href="https://i.sstatic.net/PP6IH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PP6IH.png" alt="Bellman expectation equation to calculate state value" /></a></p>
  <p>Grid(list of rows):</p>
  <pre><code>[
  
  [  0,  *,-20,-22],
  
  [-14,-18,-20,  *],
  
  [   ,-20,-18,-14],
  
  [-22,-20,   ,  0]
  
  ]
  </code></pre>
  <p>Should the * be initialized as 0 and then calculated using the equation?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-12T11:34:13.673Z
Comments.Text: Welcome to AI Stack Exchange. This is definitely solvable, but it would help if you could clarify what you mean by "the equation". We can either confirm your suggestion is correct, or show you what's wrong with it, but we cannot do either unless you show which equation you mean. There are multiple Bellman equations, so if that's what you mean, identify which one(s) you were considering using
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-12T13:28:05.303Z
Comments.Text: @NeilSlater I should have mentioned the equation. I have edited the post and added the equation now. I had this question in one of my tests and so I had copied it as it is. Even the missing values were copied as they were in question so I copied them in the same way .
Comments.UserDisplayName: ace239
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-12T13:59:06.547Z
Comments.Text: OK, so you don't need the missing values to do the calculation. They were clearly removed because they are in symmetrical positions to the `*`, so you could correctly assume the problem is symmetrical and simply copy them :-)
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><bellman-equations>
Title: Calculating state value using bellman equation question:
UserDisplayName: ace239
ThreadText: # Calculating state value using bellman equation question:
  
  (Asked by: username_0 on 2023-09-12)
  
  Consider a 4x4 grid world problem where the goal is to reach either the top left or bottom right corner. The agent can choose from four actions up,down,left,right which deterministically cause the corresponding state transitions, except that actions that would take the agent off the grid leave the state unchanged. We model this as an undiscounted,episodic task where the reward is -1 for all transitions. Suppose that the agent follows equi-probable random policy. Calculate the state values where * is placed.(Hint: Bellman expectation equation)
  
  Grid(list of rows):
  [
  
  [  0,  *,-20,-22],
  
  [-14,-18,-20,  *],
  
  [   ,-20,-18,-14],
  
  [-22,-20,   ,  0]
  
  ]
  
  Should the * be initialized as 0 and then calculated using the equation?
  
  **Tags:** reinforcement-learning, bellman-equations
  > **Comments:**
  > **username_1**: Welcome to AI Stack Exchange. This is definitely solvable, but it would help if you could clarify what you mean by "the equation". We can either confirm your suggestion is correct, or show you what's wrong with it, but we cannot do either unless you show which equation you mean. There are multiple Bellman equations, so if that's what you mean, identify which one(s) you were considering using  
  > **username_0**: @NeilSlater I should have mentioned the equation. I have edited the post and added the equation now. I had this question in one of my tests and so I had copied it as it is. Even the missing values were copied as they were in question so I copied them in the same way .  
  > **username_1**: OK, so you don't need the missing values to do the calculation. They were clearly removed because they are in symmetrical positions to the `*`, so you could correctly assume the problem is symmetrical and simply copy them :-)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-12. Score: 1 | Accepted answer
  
  > Should the \* be initialized as 0 and then calculated using the equation?
  
  This would work if you repeated the calculation multiple times, and that would be similar to a solution method called value iteration. However, it is not what is intended here. Instead, if you expand the Bellman equation for $v\_{\pi}(s)$ using your knowledge of existing values and the policy, you will get terms in your unknown on both sides (because one action in each case will stay in the same state) which you can resolve to directly find the value for the marked squares.
  
  To work through this for the `*` on the top line, we can call the value $x$ and expand the Bellman equation as follows (working clockwise from "move up" action):
  
  $$x = 0.25 \* (-1 + x) + 0.25 \* (-1 - 20) + 0.25 \* (-1 - 18 ) + 0.25 \* (-1 + 0)$$
  
  $$3x = -1 - 21 - 19 -1$$
  
  $$3x = -42$$
  
  $$x = -14$$
  
  
  ---
  
  