<blockquote>
<p>I am wondering which definition is correct.</p>
</blockquote>
<p>The asterisk * in both the definitions stands for &quot;optimal&quot; in the sense of &quot;value when following the optimal policy&quot;</p>
<p>So this one is correct:</p>
<blockquote>
<p><span class="math-container">$V^*$</span> actually assumes the optimal action in a given state, meaning <span class="math-container">$V^*$</span> would be <span class="math-container">$50$</span> in the above case</p>
</blockquote>
<p>However, you have got the definition of Q slightly wrong.</p>
<p>I think this is because you are omitting the parameters.</p>
<p>The state value function uses the state as a parameter, <span class="math-container">$V_{\pi}(s)$</span>, it returns the value of being in state <span class="math-container">$s$</span> and following a fixed policy <span class="math-container">$\pi$</span>. The * is used to denote following an optimal policy.</p>
<p>The action value function has two parameters - a state and an action that is possible in that state, <span class="math-container">$Q_{\pi}(s, a)$</span>, it returns the value of being in state <span class="math-container">$s$</span>, taking action <span class="math-container">$a$</span> (regardless of whether it is the best action or not) and following the policy <span class="math-container">$\pi$</span> <em>after</em> that point.</p>
<p>Your assertion in the question:</p>
<blockquote>
<p>And therefore my <span class="math-container">$Q^* = 50$</span></p>
</blockquote>
<p>is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You <em>could</em> say <span class="math-container">$\text{max}_a Q(\text{toy store}, a) = 50$</span>, or to choose the best action <span class="math-container">$\pi(\text{toy store}) = \text{argmax}_a Q(\text{toy store}, a) 
= \text{Lego}$</span></p>

CC BY-SA 4.0
2018-08-24T20:40:12.657Z
Ok I see.  So V* assumes the agent takes the best possible action in the state, and then follows the optimal policy thereafter.  Does that also mean the baseline at each state should be V* and that A(s, a) = 0 for the optimal case (Q(s,a) - V*), and always negative for all other cases?
Rui Nian
CC BY-SA 4.0
2018-08-24T20:48:12.377Z
@RuiNian: Yes, the maximum Advantage when you already have an optimal policy is zero.
Neil Slater
CC BY-SA 4.0
2018-08-24T20:39:00.383Z
2020-11-01T12:20:54.533Z
2020-11-01T12:20:54.533Z
Neil Slater
<p>I am confused about the definition of the optimal value (<span class="math-container">$V^*$</span>) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.</p>
<p>Originally, I thought the optimal action value, <span class="math-container">$Q^*$</span>, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.</p>
<p>And the optimal value, <span class="math-container">$V^*$</span>, being the average <span class="math-container">$Q$</span> values in that state.  Meaning that if you're in this state, the average &quot;goodness&quot; is this.</p>
<p>For example:
If I am in a toy store and I can buy a pencil, yo-yo, or Lego.</p>
<pre><code>Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50
</code></pre>
<p>And therefore my <span class="math-container">$Q^* = 50$</span></p>
<p>But my <span class="math-container">$V^*$</span> in this case is:</p>
<pre><code>V* = -10 + 5 + 50 / 3 = 15
</code></pre>
<p>Representing no matter what action I take, the average future projected reward is <span class="math-container">$15$</span>.</p>
<p>And for the advantage of learning, my baseline would be <span class="math-container">$15$</span>.  So anything less than <span class="math-container">$0$</span> is worse than average and anything above <span class="math-container">$0$</span> is better than average.</p>
<p>However, now I am reading about how <span class="math-container">$V^*$</span> actually assumes the optimal action in a given state, meaning <span class="math-container">$V^*$</span> would be 50 in the above case.</p>
<p>I am wondering which definition is correct.</p>

<reinforcement-learning><definitions><value-functions><bellman-equations>
In reinforcement learning, does the optimal value correspond to performing the best action in a given state?
Rui Nian
# In reinforcement learning, does the optimal value correspond to performing the best action in a given state?

(Asked by: username_0 on 2018-08-24)

I am confused about the definition of the optimal value ($V^*$) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.
Originally, I thought the optimal action value, $Q^*$, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.
And the optimal value, $V^*$, being the average $Q$ values in that state.  Meaning that if you're in this state, the average "goodness" is this.
For example:
If I am in a toy store and I can buy a pencil, yo-yo, or Lego.
Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50

And therefore my $Q^* = 50$
But my $V^*$ in this case is:
V* = -10 + 5 + 50 / 3 = 15

Representing no matter what action I take, the average future projected reward is $15$.
And for the advantage of learning, my baseline would be $15$.  So anything less than $0$ is worse than average and anything above $0$ is better than average.
However, now I am reading about how $V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be 50 in the above case.
I am wondering which definition is correct.

**Tags:** reinforcement-learning, definitions, value-functions, bellman-equations


---

## 1 answers

### Answer by username_1 on 2018-08-24. Score: 4 | Accepted answer

> I am wondering which definition is correct.

The asterisk \* in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"

So this one is correct:

> $V^\*$ actually assumes the optimal action in a given state, meaning $V^\*$ would be $50$ in the above case

However, you have got the definition of Q slightly wrong.

I think this is because you are omitting the parameters.

The state value function uses the state as a parameter, $V\_{\pi}(s)$, it returns the value of being in state $s$ and following a fixed policy $\pi$. The \* is used to denote following an optimal policy.

The action value function has two parameters - a state and an action that is possible in that state, $Q\_{\pi}(s, a)$, it returns the value of being in state $s$, taking action $a$ (regardless of whether it is the best action or not) and following the policy $\pi$ *after* that point.

Your assertion in the question:

> And therefore my $Q^\* = 50$

is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You *could* say $\text{max}\_a Q(\text{toy store}, a) = 50$, or to choose the best action $\pi(\text{toy store}) = \text{argmax}\_a Q(\text{toy store}, a)
= \text{Lego}$

> **Comments:**
> **username_0**: Ok I see.  So V* assumes the agent takes the best possible action in the state, and then follows the optimal policy thereafter.  Does that also mean the baseline at each state should be V* and that A(s, a) = 0 for the optimal case (Q(s,a) - V*), and always negative for all other cases?  
> **username_1**: @RuiNian: Yes, the maximum Advantage when you already have an optimal policy is zero.  

---

