Answers.Body: <p>I am not sure your equations are correct or I understand very well your question but I write the main equations of <span class="math-container">$V$</span> and <span class="math-container">$Q$</span> functions from Sutton book that may help you:</p>
  <p><span class="math-container">$ v^{*}(s) = max v_{\pi}(s) $</span></p>
  <p><span class="math-container">$ q^{*}(s,a) = max q_{\pi}(s,a) $</span></p>
  <p><span class="math-container">$  v^{*}(s)=max\mathbb{E}[R_{t+1}+\gamma v^{*}(s+1)|S_{t}=s, A_{t}=a] $</span></p>
  <p><span class="math-container">$  q^{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma max q^{*}(s+1,a^{'})|S_{t}=s, A_{t}=a] $</span></p>
  <p><span class="math-container">$  v^{*}(s)=max q^{*}_{\pi}(s,a) $</span></p>
  <p>As you can see, the optimal value function is equal to the maximum Q function on all feasible actions. We will reach optimal policy if we calculate the <span class="math-container">$argmax$</span> of these functions on possible actions. Also, the policy found by the value function is not always the same as the Q function because their features are different. The Q function considers both actions and states that not only decrease the speed of the process but also choose the shortest optimal path. For example, you can test simple graphical navigation on below Github repository:
  <a href="https://github.com/pouyan-asg/global-path-planning" rel="nofollow noreferrer">https://github.com/pouyan-asg/global-path-planning</a></p>
  <p>In this example, you will see the path that agent chooses to reach the destination is not similar in both cases, and the timing in value function mode is higher (because the agent does not consider the action). I hope my answer is clear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T14:55:41.487Z
Answers.Text: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T14:21:45.070Z
Answers.LastActivityDate: 2022-07-07T15:23:17.487Z
Answers.LastEditDate: 2022-07-07T15:23:17.487Z
Answers.UserDisplayName: Pouyan
Body: <p>In reinforcement learning, we define the optimal policy <span class="math-container">$\pi^*$</span> as the policy that maximizes the value of the state:</p>
  <p><span class="math-container">$$
  \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}
  $$</span></p>
  <p>In Q-learning, we try to find a policy that maximize the state-action value function Q:</p>
  <p><span class="math-container">$$
  \pi_q^*=\underset{\pi}{\operatorname{argmax}} {Q_{\pi}(s,a)}
  $$</span></p>
  <p>However, does maximizing the value function and maximizing the state-action value function generate the same optimal policy? In the general case of continuous, stochastic action, <span class="math-container">$V_{\pi}(s)$</span> is connected to <span class="math-container">$Q_{\pi}(s,a)$</span> by:</p>
  <p><span class="math-container">$$
  V_{\pi}(s)=E_{a\sim\pi}[Q_{\pi}(s,a)]
  $$</span></p>
  <p>So</p>
  <p><span class="math-container">$$
  \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}=\underset{\pi}{\operatorname{argmax}} E_{a\sim\pi}[Q_{\pi}(s,a)]
  $$</span></p>
  <p>And if <span class="math-container">$\pi_v^*=\pi_q^*$</span>, mathematically I'm not sure why the expectation <span class="math-container">$E_{a\sim\pi}$</span> before <span class="math-container">$Q_{\pi}(s,a)$</span> can be simply dropped.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T08:24:42.090Z
Comments.Text: "we try to find a policy that maximize the state-action value function Q", really? Or are we trying to find a policy that is assumed to be greedy wrt to the optimal state-action function? In an MDP, there's a unique optimal value function  and state-action value function . I am not sure I understand your question or confusion. It seems that you're mixing the theory (mathematical results) with the practice (algorithms). However, asking if two algorithms lead to the same _optimal policy_ is reasonable because there can be more than one. So, can you clarify what you're really asking here?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T09:04:28.740Z
Comments.Text: Your first equation has a problem - an optimal policy is not the one that maximizes the value of *a particular state $s$* - rather it maximizes the value of *any* state. The second equation has the same problem, along with having $a$ is unspecified - here, you should take the expectation w.r.t $\pi$.
Comments.UserDisplayName: mikkola
Tags: <reinforcement-learning><q-learning><markov-decision-process><value-functions>
Title: Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?
UserDisplayName: Cloudy
ThreadText: # Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?
  
  (Asked by: username_0 on 2022-07-02)
  
  In reinforcement learning, we define the optimal policy $\pi^\*$ as the policy that maximizes the value of the state:
  
  $$
  \pi\_v^\*=\underset{\pi}{\operatorname{argmax}} {V\_{\pi}(s)}
  $$
  
  In Q-learning, we try to find a policy that maximize the state-action value function Q:
  
  $$
  \pi\_q^\*=\underset{\pi}{\operatorname{argmax}} {Q\_{\pi}(s,a)}
  $$
  
  However, does maximizing the value function and maximizing the state-action value function generate the same optimal policy? In the general case of continuous, stochastic action, $V\_{\pi}(s)$ is connected to $Q\_{\pi}(s,a)$ by:
  
  $$
  V\_{\pi}(s)=E\_{a\sim\pi}[Q\_{\pi}(s,a)]
  $$
  
  So
  
  $$
  \pi\_v^\*=\underset{\pi}{\operatorname{argmax}} {V\_{\pi}(s)}=\underset{\pi}{\operatorname{argmax}} E\_{a\sim\pi}[Q\_{\pi}(s,a)]
  $$
  
  And if $\pi\_v^\*=\pi\_q^\*$, mathematically I'm not sure why the expectation $E\_{a\sim\pi}$ before $Q\_{\pi}(s,a)$ can be simply dropped.
  
  **Tags:** reinforcement-learning, q-learning, markov-decision-process, value-functions
  > **Comments:**
  > **username_1**: "we try to find a policy that maximize the state-action value function Q", really? Or are we trying to find a policy that is assumed to be greedy wrt to the optimal state-action function? In an MDP, there's a unique optimal value function  and state-action value function . I am not sure I understand your question or confusion. It seems that you're mixing the theory (mathematical results) with the practice (algorithms). However, asking if two algorithms lead to the same _optimal policy_ is reasonable because there can be more than one. So, can you clarify what you're really asking here?  
  > **username_2**: Your first equation has a problem - an optimal policy is not the one that maximizes the value of *a particular state $s$* - rather it maximizes the value of *any* state. The second equation has the same problem, along with having $a$ is unspecified - here, you should take the expectation w.r.t $\pi$.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-07-07. Score: 0
  
  I am not sure your equations are correct or I understand very well your question but I write the main equations of $V$ and $Q$ functions from Sutton book that may help you:
  
  $ v^{\*}(s) = max v\_{\pi}(s) $
  
  $ q^{\*}(s,a) = max q\_{\pi}(s,a) $
  
  $ v^{\*}(s)=max\mathbb{E}[R\_{t+1}+\gamma v^{\*}(s+1)|S\_{t}=s, A\_{t}=a] $
  
  $ q^{\*}(s,a)=\mathbb{E}[R\_{t+1}+\gamma max q^{\*}(s+1,a^{'})|S\_{t}=s, A\_{t}=a] $
  
  $ v^{\*}(s)=max q^{\*}\_{\pi}(s,a) $
  
  As you can see, the optimal value function is equal to the maximum Q function on all feasible actions. We will reach optimal policy if we calculate the $argmax$ of these functions on possible actions. Also, the policy found by the value function is not always the same as the Q function because their features are different. The Q function considers both actions and states that not only decrease the speed of the process but also choose the shortest optimal path. For example, you can test simple graphical navigation on below Github repository:
  <https://github.com/pouyan-asg/global-path-planning>
  
  In this example, you will see the path that agent chooses to reach the destination is not similar in both cases, and the timing in value function mode is higher (because the agent does not consider the action). I hope my answer is clear.
  
  > **Comments:**
  > **username_4**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  