Answers.Body: <p>The <strong>definition</strong> of the state-action value function is always the same.</p>
  <p>Your definition is correct, as <span class="math-container">$q_{\pi}(s,a)$</span> is conditioned on <span class="math-container">$a$</span>, so you don't need to write <span class="math-container">$q_{\pi}(s,a)$</span> as an conditional expectation that depends on <span class="math-container">$\pi$</span>. In fact, the conditional expectation is taken wrt the probability distribution <span class="math-container">$p(s',r|s,a)$</span>. However, you need the subscript <span class="math-container">$\pi$</span> in <span class="math-container">$v_\pi(s')$</span> because <span class="math-container">$v_\pi(s')$</span> is defined as the expected return by following <span class="math-container">$\pi$</span> starting in <span class="math-container">$s'$</span>.</p>
  <p>If you didn't write <span class="math-container">$q_{\pi}(s,a)$</span> in terms of <span class="math-container">$v_\pi(s')$</span>, then you could write <span class="math-container">$q_{\pi}(s,a)$</span> as an expectation that depends on <span class="math-container">$\pi$</span>, because, in that case, <span class="math-container">$q_{\pi}(s,a)$</span>, is defined as an expectation of the return (after having taken <span class="math-container">$a$</span> in <span class="math-container">$s$</span>), which depends on <span class="math-container">$\pi$</span> (see equation 3.3 of <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=80" rel="nofollow noreferrer">Sutton &amp; Barto book</a>, p. 58). Of course, this way of writing <span class="math-container">$q_{\pi}(s,a)$</span> is equivalent to writing it in terms of <span class="math-container">$v_\pi(s')$</span>.</p>
  <p>I think David Silver's notation might be an abuse of notation. In his equation, the policy is parametrized by <span class="math-container">$\theta$</span>, so I think he wants to emphasize that you will estimate the state-action value function (critic) based on <span class="math-container">$\pi_\theta$</span> (actor). Alternatively, he uses <span class="math-container">$\pi_\theta$</span> as a subscript of <span class="math-container">$\mathbb{E}_{\pi_\theta}$</span> to emphasize that the future return starting in <span class="math-container">$s'$</span>, after having taken action <span class="math-container">$a$</span> in <span class="math-container">$s$</span>, still depends on <span class="math-container">$\pi_\theta$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-07T13:13:04.883Z
Answers.LastActivityDate: 2022-01-24T13:20:48.917Z
Answers.LastEditDate: 2022-01-24T13:20:48.917Z
Answers.UserDisplayName: nbro
Body: <p>I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function written in terms of the value function. </p>
  
  <p>From Question 3.13 of the textbook I am able to write the state-action value function as
  <span class="math-container">$$q_{\pi}(s,a) = \sum_{s',r}p(s',r|s,a)(r + \gamma v_\pi(s')) = \mathbb{E}[r + \gamma v_\pi(s')|s,a]\;.$$</span>
  Note that the expectation is <em>not</em> taken with respect to <span class="math-container">$\pi$</span> as <span class="math-container">$\pi$</span> is the conditional probability of taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. Now, in David Silver's slides for the Actor-Critic methods of the Policy Gradient lectures, he says that 
  <span class="math-container">$$\mathbb{E}_{\pi_\theta}[r + \gamma v_{\pi_\theta}(s')|s,a] = q_{\pi_\theta}(s,a)\;.$$</span></p>
  
  <p>Are these two definitions equivalent (in expectation)? </p>
  
Tags: <reinforcement-learning><comparison><value-functions><expectation><bellman-equations>
Title: Are these two definitions of the state-action value function equivalent?
UserDisplayName: David
ThreadText: # Are these two definitions of the state-action value function equivalent?
  
  (Asked by: username_0 on 2020-05-07)
  
  I have been reading the Sutton and Barto textbook and going through username_0 Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function written in terms of the value function.
  
  From Question 3.13 of the textbook I am able to write the state-action value function as
  $$q\_{\pi}(s,a) = \sum\_{s',r}p(s',r|s,a)(r + \gamma v\_\pi(s')) = \mathbb{E}[r + \gamma v\_\pi(s')|s,a]\;.$$
  Note that the expectation is *not* taken with respect to $\pi$ as $\pi$ is the conditional probability of taking action $a$ in state $s$. Now, in username_0 Silver's slides for the Actor-Critic methods of the Policy Gradient lectures, he says that
  $$\mathbb{E}\_{\pi\_\theta}[r + \gamma v\_{\pi\_\theta}(s')|s,a] = q\_{\pi\_\theta}(s,a)\;.$$
  
  Are these two definitions equivalent (in expectation)?
  
  **Tags:** reinforcement-learning, comparison, value-functions, expectation, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-07. Score: 3 | Accepted answer
  
  The **definition** of the state-action value function is always the same.
  
  Your definition is correct, as $q\_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q\_{\pi}(s,a)$ as an conditional expectation that depends on $\pi$. In fact, the conditional expectation is taken wrt the probability distribution $p(s',r|s,a)$. However, you need the subscript $\pi$ in $v\_\pi(s')$ because $v\_\pi(s')$ is defined as the expected return by following $\pi$ starting in $s'$.
  
  If you didn't write $q\_{\pi}(s,a)$ in terms of $v\_\pi(s')$, then you could write $q\_{\pi}(s,a)$ as an expectation that depends on $\pi$, because, in that case, $q\_{\pi}(s,a)$, is defined as an expectation of the return (after having taken $a$ in $s$), which depends on $\pi$ (see equation 3.3 of [Sutton & Barto book](http://incompleteideas.net/book/RLbook2020.pdf#page=80), p. 58). Of course, this way of writing $q\_{\pi}(s,a)$ is equivalent to writing it in terms of $v\_\pi(s')$.
  
  I think username_0 Silver's notation might be an abuse of notation. In his equation, the policy is parametrized by $\theta$, so I think he wants to emphasize that you will estimate the state-action value function (critic) based on $\pi\_\theta$ (actor). Alternatively, he uses $\pi\_\theta$ as a subscript of $\mathbb{E}\_{\pi\_\theta}$ to emphasize that the future return starting in $s'$, after having taken action $a$ in $s$, still depends on $\pi\_\theta$.
  
  
  ---
  
  