<p>Nevermind. I found that above answer is indeed correct, but the gradescope has a bug (it requires the format to be .2 instead of 0.2).</p>

CC BY-SA 4.0
2020-02-17T05:25:40.757Z
2020-02-17T05:25:40.757Z
libphy
<p>I've been following the Berkeley cs188's assignment (I'm not taking the course). Currently, they don't show the solution in the gradescope unless I get it correct. 
<a href="https://i.sstatic.net/uMTm4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uMTm4.png" alt="enter image description here"></a></p>

<p>My reasoning was</p>

<p><span class="math-container">$V^*(a)$</span> = 10 fixed, because the optimal action is to terminate and receive the reward 10. </p>

<p><span class="math-container">$V^*(b) = 10 \times 0.2 = 2$</span> using Bellman optimality eqn <span class="math-container">$V^*(s) = R(s)+ \gamma \ \rm{max}_{a} \sum_{s'} P(s'|s,a) V^*(s')$</span>, where the optimal actionfrom b is to left.</p>

<p>Similarly, I get <span class="math-container">$V^*(c) = 10 \times (0.2)^2 = 0.4$</span></p>

<p>For the state <span class="math-container">$d$</span>, it is optimal to move to the right and exit at <span class="math-container">$e$</span> to receive 1, therefore <span class="math-container">$V^*(d) = 1 \times 0.2 = 0.2$</span>.</p>

<p>And <span class="math-container">$V^*(e) = 1$</span> fixed. </p>

<p>However, there autograder says it's incorrect and doesn't show explanation. Can anyone explain what the right approach or answer is?  </p>

<markov-decision-process><value-iteration>
Unable to understand V* at infinite time horizon using Bellman equation for solving MDP
libphy
# Unable to understand V* at infinite time horizon using Bellman equation for solving MDP

(Asked by: username_0 on 2020-02-17)

I've been following the Berkeley cs188's assignment (I'm not taking the course). Currently, they don't show the solution in the gradescope unless I get it correct.
[![enter image description here](https://i.sstatic.net/uMTm4.png)](https://i.sstatic.net/uMTm4.png)

My reasoning was

$V^\*(a)$ = 10 fixed, because the optimal action is to terminate and receive the reward 10.

$V^\*(b) = 10 \times 0.2 = 2$ using Bellman optimality eqn $V^\*(s) = R(s)+ \gamma \ \rm{max}\_{a} \sum\_{s'} P(s'|s,a) V^\*(s')$, where the optimal actionfrom b is to left.

Similarly, I get $V^\*(c) = 10 \times (0.2)^2 = 0.4$

For the state $d$, it is optimal to move to the right and exit at $e$ to receive 1, therefore $V^\*(d) = 1 \times 0.2 = 0.2$.

And $V^\*(e) = 1$ fixed.

However, there autograder says it's incorrect and doesn't show explanation. Can anyone explain what the right approach or answer is?

**Tags:** markov-decision-process, value-iteration


---

## 1 answers

### Answer by username_0 on 2020-02-17. Score: 1

Nevermind. I found that above answer is indeed correct, but the gradescope has a bug (it requires the format to be .2 instead of 0.2).


---

