Answers.Body: <blockquote>
  <p>I am wondering which definition is correct.</p>
  </blockquote>
  <p>The asterisk * in both the definitions stands for &quot;optimal&quot; in the sense of &quot;value when following the optimal policy&quot;</p>
  <p>So this one is correct:</p>
  <blockquote>
  <p><span class="math-container">$V^*$</span> actually assumes the optimal action in a given state, meaning <span class="math-container">$V^*$</span> would be <span class="math-container">$50$</span> in the above case</p>
  </blockquote>
  <p>However, you have got the definition of Q slightly wrong.</p>
  <p>I think this is because you are omitting the parameters.</p>
  <p>The state value function uses the state as a parameter, <span class="math-container">$V_{\pi}(s)$</span>, it returns the value of being in state <span class="math-container">$s$</span> and following a fixed policy <span class="math-container">$\pi$</span>. The * is used to denote following an optimal policy.</p>
  <p>The action value function has two parameters - a state and an action that is possible in that state, <span class="math-container">$Q_{\pi}(s, a)$</span>, it returns the value of being in state <span class="math-container">$s$</span>, taking action <span class="math-container">$a$</span> (regardless of whether it is the best action or not) and following the policy <span class="math-container">$\pi$</span> <em>after</em> that point.</p>
  <p>Your assertion in the question:</p>
  <blockquote>
  <p>And therefore my <span class="math-container">$Q^* = 50$</span></p>
  </blockquote>
  <p>is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You <em>could</em> say <span class="math-container">$\text{max}_a Q(\text{toy store}, a) = 50$</span>, or to choose the best action <span class="math-container">$\pi(\text{toy store}) = \text{argmax}_a Q(\text{toy store}, a) 
  = \text{Lego}$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-24T20:40:12.657Z
Answers.Text: Ok I see.  So V* assumes the agent takes the best possible action in the state, and then follows the optimal policy thereafter.  Does that also mean the baseline at each state should be V* and that A(s, a) = 0 for the optimal case (Q(s,a) - V*), and always negative for all other cases?
Answers.UserDisplayName: Rui Nian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-24T20:48:12.377Z
Answers.Text: @RuiNian: Yes, the maximum Advantage when you already have an optimal policy is zero.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-24T20:39:00.383Z
Answers.LastActivityDate: 2020-11-01T12:20:54.533Z
Answers.LastEditDate: 2020-11-01T12:20:54.533Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am confused about the definition of the optimal value (<span class="math-container">$V^*$</span>) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.</p>
  <p>Originally, I thought the optimal action value, <span class="math-container">$Q^*$</span>, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.</p>
  <p>And the optimal value, <span class="math-container">$V^*$</span>, being the average <span class="math-container">$Q$</span> values in that state.  Meaning that if you're in this state, the average &quot;goodness&quot; is this.</p>
  <p>For example:
  If I am in a toy store and I can buy a pencil, yo-yo, or Lego.</p>
  <pre><code>Q(toy store, pencil) = -10
  Q(toy store, yo-yo) = 5
  Q(toy store, Lego) = 50
  </code></pre>
  <p>And therefore my <span class="math-container">$Q^* = 50$</span></p>
  <p>But my <span class="math-container">$V^*$</span> in this case is:</p>
  <pre><code>V* = -10 + 5 + 50 / 3 = 15
  </code></pre>
  <p>Representing no matter what action I take, the average future projected reward is <span class="math-container">$15$</span>.</p>
  <p>And for the advantage of learning, my baseline would be <span class="math-container">$15$</span>.  So anything less than <span class="math-container">$0$</span> is worse than average and anything above <span class="math-container">$0$</span> is better than average.</p>
  <p>However, now I am reading about how <span class="math-container">$V^*$</span> actually assumes the optimal action in a given state, meaning <span class="math-container">$V^*$</span> would be 50 in the above case.</p>
  <p>I am wondering which definition is correct.</p>
  
Tags: <reinforcement-learning><definitions><value-functions><bellman-equations>
Title: In reinforcement learning, does the optimal value correspond to performing the best action in a given state?
UserDisplayName: Rui Nian
ThreadText: # In reinforcement learning, does the optimal value correspond to performing the best action in a given state?
  
  (Asked by: username_0 on 2018-08-24)
  
  I am confused about the definition of the optimal value ($V^\*$) and optimal action-value (Q\*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.
  
  Originally, I thought the optimal action value, $Q^\*$, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.
  
  And the optimal value, $V^\*$, being the average $Q$ values in that state. Meaning that if you're in this state, the average "goodness" is this.
  
  For example:
  If I am in a toy store and I can buy a pencil, yo-yo, or Lego.
  
  ```
  Q(toy store, pencil) = -10
  Q(toy store, yo-yo) = 5
  Q(toy store, Lego) = 50
  
  ```
  
  And therefore my $Q^\* = 50$
  
  But my $V^\*$ in this case is:
  
  ```
  V* = -10 + 5 + 50 / 3 = 15
  
  ```
  
  Representing no matter what action I take, the average future projected reward is $15$.
  
  And for the advantage of learning, my baseline would be $15$. So anything less than $0$ is worse than average and anything above $0$ is better than average.
  
  However, now I am reading about how $V^\*$ actually assumes the optimal action in a given state, meaning $V^\*$ would be 50 in the above case.
  
  I am wondering which definition is correct.
  
  **Tags:** reinforcement-learning, definitions, value-functions, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-24. Score: 4 | Accepted answer
  
  > I am wondering which definition is correct.
  
  The asterisk \* in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"
  
  So this one is correct:
  
  > $V^\*$ actually assumes the optimal action in a given state, meaning $V^\*$ would be $50$ in the above case
  
  However, you have got the definition of Q slightly wrong.
  
  I think this is because you are omitting the parameters.
  
  The state value function uses the state as a parameter, $V\_{\pi}(s)$, it returns the value of being in state $s$ and following a fixed policy $\pi$. The \* is used to denote following an optimal policy.
  
  The action value function has two parameters - a state and an action that is possible in that state, $Q\_{\pi}(s, a)$, it returns the value of being in state $s$, taking action $a$ (regardless of whether it is the best action or not) and following the policy $\pi$ *after* that point.
  
  Your assertion in the question:
  
  > And therefore my $Q^\* = 50$
  
  is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You *could* say $\text{max}\_a Q(\text{toy store}, a) = 50$, or to choose the best action $\pi(\text{toy store}) = \text{argmax}\_a Q(\text{toy store}, a)
  = \text{Lego}$
  
  > **Comments:**
  > **username_0**: Ok I see.  So V* assumes the agent takes the best possible action in the state, and then follows the optimal policy thereafter.  Does that also mean the baseline at each state should be V* and that A(s, a) = 0 for the optimal case (Q(s,a) - V*), and always negative for all other cases?  
  > **username_1**: @username_0: Yes, the maximum Advantage when you already have an optimal policy is zero.  
  
  ---
  
  