Answers.Body: <blockquote>
  <p>This is the maximum value achievable under any policy</p>
  </blockquote>
  <p>The statement says that <span class="math-container">$Q^*(s,a)$</span> is the maximum possible state-action value that is achievable (at most) by any policy, <span class="math-container">$\pi$</span>: including the random policy, and the optimal policy. It is also explained in slide 23:</p>
  <ul>
  <li>&quot;An optimal value function is the maximum achievable value&quot;</li>
  <li>&quot;Once you have <span class="math-container">$Q^*$</span> we can act optimally&quot;</li>
  <li>&quot;Optimal value maximizes over all decisions&quot;</li>
  </ul>
  <p>Indeed, the optimal policy <span class="math-container">$\pi^*$</span> would always achieve <span class="math-container">$Q^*$</span>. It does not mean that the implicit policy (the one that can be derived from it: <span class="math-container">$\pi^*(s)=\arg\max_aQ^*(s,a)$</span>) is not important, neither that it cannot be computed from <span class="math-container">$Q^*$</span>.</p>
  <p>You have to think about this in terms of <em>policy evaluation</em> (or <em>prediction</em>, in classical RL terminology): you have the optimal action-value table, that you can use to compute the value (or return) that some policy achieves, as well as to derive the optimal policy by taking the action that maximizes <span class="math-container">$Q^*$</span> for a given state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T20:35:15.930Z
Answers.LastActivityDate: 2023-05-31T20:35:15.930Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>The slide 22 in your link says:</p>
  <blockquote>
  <ul>
  <li>Once we have <span class="math-container">$Q^*$</span> we can act optimally:
  <span class="math-container">$$\pi^*(s) = \arg \max_a Q^*(s,a)$$</span></li>
  </ul>
  </blockquote>
  <p>That is the optimal policy, which is greedy wrt <span class="math-container">$Q^*$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T10:57:18.667Z
Answers.Text: Yeah, but there comes my confusion with the statement in slide 27. It then says "... under any policy". Does this mean this derived optimal policy is irrelevant? Does "any policy" here mean that we don't need the optimal policy to obtain the expected maximum value?
Answers.UserDisplayName: mathnoob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-30T11:11:27.203Z
Answers.LastActivityDate: 2023-05-30T11:11:27.203Z
Answers.UserDisplayName: Kostya
Body: <p>In slide <a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/deep_rl_tutorial_small_compressed.pdf" rel="nofollow noreferrer">27/66</a>, it states that &quot;<strong>This is the maximum value achievable under any policy</strong>&quot;. My understanding of Value-based RL is that the optimal Q-table/function will be learnt. Afterwards, the implicit optimal policy can be &quot;derived&quot; by greedily picking the actions based on this learnt Q-value.</p>
  <p>Does the implicit policy not matter here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-30T15:07:45.737Z
Comments.Text: Probably related: https://ai.stackexchange.com/questions/40540/how-do-we-get-the-optimal-value-function
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-31T22:42:05.953Z
Comments.Text: I have not idea what you're asking here. The implicit policy, which you don't define, is supposed to matter to what? Reformulate the question in a more specific and clearer way, please.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning>
Title: Conceptual clarification on Value-based Reinforcement Learning
UserDisplayName: mathnoob
ThreadText: # Conceptual clarification on Value-based Reinforcement Learning
  
  (Asked by: username_0 on 2023-05-30)
  
  In slide [27/66](https://www.davidsilver.uk/wp-content/uploads/2020/03/deep_rl_tutorial_small_compressed.pdf), it states that "**This is the maximum value achievable under any policy**". My understanding of Value-based RL is that the optimal Q-table/function will be learnt. Afterwards, the implicit optimal policy can be "derived" by greedily picking the actions based on this learnt Q-value.
  
  Does the implicit policy not matter here?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Probably related: https://ai.stackexchange.com/questions/40540/how-do-we-get-the-optimal-value-function  
  > **username_2**: I have not idea what you're asking here. The implicit policy, which you don't define, is supposed to matter to what? Reformulate the question in a more specific and clearer way, please.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-05-31. Score: 0 | Accepted answer
  
  > This is the maximum value achievable under any policy
  
  The statement says that $Q^\*(s,a)$ is the maximum possible state-action value that is achievable (at most) by any policy, $\pi$: including the random policy, and the optimal policy. It is also explained in slide 23:
  
  * "An optimal value function is the maximum achievable value"
  * "Once you have $Q^\*$ we can act optimally"
  * "Optimal value maximizes over all decisions"
  
  Indeed, the optimal policy $\pi^\*$ would always achieve $Q^\*$. It does not mean that the implicit policy (the one that can be derived from it: $\pi^\*(s)=\arg\max\_aQ^\*(s,a)$) is not important, neither that it cannot be computed from $Q^\*$.
  
  You have to think about this in terms of *policy evaluation* (or *prediction*, in classical RL terminology): you have the optimal action-value table, that you can use to compute the value (or return) that some policy achieves, as well as to derive the optimal policy by taking the action that maximizes $Q^\*$ for a given state.
  
  
  ---
  
  ### Answer by username_3 on 2023-05-30. Score: 1
  
  The slide 22 in your link says:
  
  > * Once we have $Q^\*$ we can act optimally:
  >   $$\pi^\*(s) = \arg \max\_a Q^\*(s,a)$$
  
  That is the optimal policy, which is greedy wrt $Q^\*$.
  
  > **Comments:**
  > **username_0**: Yeah, but there comes my confusion with the statement in slide 27. It then says "... under any policy". Does this mean this derived optimal policy is irrelevant? Does "any policy" here mean that we don't need the optimal policy to obtain the expected maximum value?  
  
  ---
  
  