Answers.Body: <p>Nevermind. I found that above answer is indeed correct, but the gradescope has a bug (it requires the format to be .2 instead of 0.2).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-17T05:25:40.757Z
Answers.LastActivityDate: 2020-02-17T05:25:40.757Z
Answers.UserDisplayName: libphy
Body: <p>I've been following the Berkeley cs188's assignment (I'm not taking the course). Currently, they don't show the solution in the gradescope unless I get it correct. 
  <a href="https://i.sstatic.net/uMTm4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uMTm4.png" alt="enter image description here"></a></p>
  
  <p>My reasoning was</p>
  
  <p><span class="math-container">$V^*(a)$</span> = 10 fixed, because the optimal action is to terminate and receive the reward 10. </p>
  
  <p><span class="math-container">$V^*(b) = 10 \times 0.2 = 2$</span> using Bellman optimality eqn <span class="math-container">$V^*(s) = R(s)+ \gamma \ \rm{max}_{a} \sum_{s'} P(s'|s,a) V^*(s')$</span>, where the optimal actionfrom b is to left.</p>
  
  <p>Similarly, I get <span class="math-container">$V^*(c) = 10 \times (0.2)^2 = 0.4$</span></p>
  
  <p>For the state <span class="math-container">$d$</span>, it is optimal to move to the right and exit at <span class="math-container">$e$</span> to receive 1, therefore <span class="math-container">$V^*(d) = 1 \times 0.2 = 0.2$</span>.</p>
  
  <p>And <span class="math-container">$V^*(e) = 1$</span> fixed. </p>
  
  <p>However, there autograder says it's incorrect and doesn't show explanation. Can anyone explain what the right approach or answer is?  </p>
  
Tags: <markov-decision-process><value-iteration>
Title: Unable to understand V* at infinite time horizon using Bellman equation for solving MDP
UserDisplayName: libphy
ThreadText: # Unable to understand V* at infinite time horizon using Bellman equation for solving MDP
  
  (Asked by: username_0 on 2020-02-17)
  
  I've been following the Berkeley cs188's assignment (I'm not taking the course). Currently, they don't show the solution in the gradescope unless I get it correct.
  [![enter image description here](https://i.sstatic.net/uMTm4.png)](https://i.sstatic.net/uMTm4.png)
  
  My reasoning was
  
  $V^\*(a)$ = 10 fixed, because the optimal action is to terminate and receive the reward 10.
  
  $V^\*(b) = 10 \times 0.2 = 2$ using Bellman optimality eqn $V^\*(s) = R(s)+ \gamma \ \rm{max}\_{a} \sum\_{s'} P(s'|s,a) V^\*(s')$, where the optimal actionfrom b is to left.
  
  Similarly, I get $V^\*(c) = 10 \times (0.2)^2 = 0.4$
  
  For the state $d$, it is optimal to move to the right and exit at $e$ to receive 1, therefore $V^\*(d) = 1 \times 0.2 = 0.2$.
  
  And $V^\*(e) = 1$ fixed.
  
  However, there autograder says it's incorrect and doesn't show explanation. Can anyone explain what the right approach or answer is?
  
  **Tags:** markov-decision-process, value-iteration
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-02-17. Score: 1
  
  Nevermind. I found that above answer is indeed correct, but the gradescope has a bug (it requires the format to be .2 instead of 0.2).
  
  
  ---
  
  