Answers.Body: <p>There is also the simpler <strong>action value function</strong> <span class="math-container">$$q_*(a) = \mathbb{E} \left[ R_t 
   \mid A_t = a\right],$$</span> which we try to approximate when solving <em>context-free</em> <strong>bandit problems</strong>. You can also similarly define the action value function for <em>contextual</em> bandit problems by also conditioning on the context (rather than just on the action).</p>
  <p>See chapter 2 of the book <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=48" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> (2nd edition) by Barto and Sutton for more details.</p>
  <p>There are also the <strong>afterstate value functions</strong>, <span class="math-container">$v(s')$</span>. Check <a href="https://ai.stackexchange.com/q/24816/2444">this question</a> and <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=158" rel="nofollow noreferrer">section 6.8</a> of the just cited book.</p>
  <p>Moreover, there's the <strong>state-action-goal value function</strong>, <span class="math-container">$q(s, a, g)$</span>, as described in the paper <a href="https://papers.nips.cc/paper/2017/file/453fadbd8a1a3af50a9df4df899537b5-Paper.pdf" rel="nofollow noreferrer">Hindsight Experience Replay</a> (2017), although I am not sure how we can mathematically define it (i.e. as a function of other value functions or as a Bellman equation).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-29T11:28:03.113Z
Answers.LastActivityDate: 2020-11-23T14:13:36.230Z
Answers.LastEditDate: 2020-11-23T14:13:36.230Z
Answers.UserDisplayName: nbro
Answers.Body: <p><strong>Advantage function</strong>: <span class="math-container">$A(s,a) = Q(s,a) - V(s)$</span></p>
  <p>More interesting is the <strong>General Value Function (GVF)</strong>, the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function <span class="math-container">$V(s)$</span>. The GVF is defined on page 459 of the 2nd edition of <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton and Barto's RL book</a> as
  <span class="math-container">$$v_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum_{k=t}^\infty\left(\prod_{i=t+1}^k \gamma(S_i)\right)C_{k+1}\right\rvert S_t=s, A_{t:\infty}\sim\pi\right]$$</span>
  where <span class="math-container">$C_t \in \mathbb{R}$</span> is the signal being summed over time.</p>
  <p><span class="math-container">$\gamma(S_t)$</span> is a function <span class="math-container">$\gamma: \cal{S}\to[0,1]$</span> allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. <a href="http://reinforcement-learning.ml/papers/pgmrl2018_sherstan.pdf" rel="nofollow noreferrer">Some</a> call it the continuation function.</p>
  <p>Also of note are the <strong>differential value functions</strong>. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward <span class="math-container">$R_{t+1}-r(\pi)$</span>, where <span class="math-container">$r(\pi)$</span> is the average reward under policy <span class="math-container">$\pi$</span>.</p>
  <p><span class="math-container">$$v_{\pi,\,diff}(s) = \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v_{\pi,\,diff}(s')\right]$$</span>
  <span class="math-container">$$v_{*,\,diff}(s) = \max_a \sum_{s',r} p(s',r|s,a)\left[r-\max_\pi r(\pi)+ v_{*,\,diff}(s')\right]$$</span></p>
  <p>The differential value functions assume that a single fixed value of <span class="math-container">$r(\pi)$</span> exists. That is, they assume the MDP is &quot;ergodic.&quot; See section 10.3 of Sutton and Barto for details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-06T16:02:05.983Z
Answers.Text: Regarding the "Some" link: This site [http://reinforcement-learning.ml](http://reinforcement-learning.ml), I assume a previous github pages size, is offline.
Answers.UserDisplayName: Nikolaj-K
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T02:00:16.780Z
Answers.LastActivityDate: 2020-11-23T14:03:51.547Z
Answers.LastEditDate: 2020-11-23T14:03:51.547Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>In reinforcement learning, we often define two functions, the <em>state-value function</em></p>
  
  <p><span class="math-container">$$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} 
  \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$</span></p>
  
  <p>and the <em>state-action-value function</em></p>
  
  <p><span class="math-container">$$Q^\pi(s,a) = \mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}\Bigg|S_t=s, A_t=a \right]$$</span></p>
  
  <p>where <span class="math-container">$\mathbb{E}_{\pi}$</span> means that these functions are defined as the <a href="https://en.wikipedia.org/wiki/Expected_value" rel="nofollow noreferrer">expectation</a> with respect to a fixed policy <span class="math-container">$\pi$</span> of what is often called the <em>return</em>, <span class="math-container">$\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1}$</span>, where <span class="math-container">$\gamma$</span> is a <em>discount factor</em> and <span class="math-container">$R_{t+k+1}$</span> is the <em>reward</em> received from the environment (while the agent interacts with it) from time <span class="math-container">$t$</span> onwards. </p>
  
  <p>So, both the <span class="math-container">$V$</span> and <span class="math-container">$Q$</span> functions are defined as expectations of the return (or the <em>cumulative future discounted reward</em>), but these expectations have different "conditions" (or are conditioned on different variables). The <span class="math-container">$V$</span> function is the expectation (with respect to a fixed policy <span class="math-container">$\pi$</span>) of the return given that the current state (the state at time <span class="math-container">$t$</span>) is <span class="math-container">$s$</span>. The <span class="math-container">$Q$</span> function is the expectation (with respect to a fixed policy <span class="math-container">$\pi$</span>) of the return conditioned on the fact that the current state the agent is in is <span class="math-container">$s$</span> and the action the agent takes at <span class="math-container">$s$</span> is <span class="math-container">$a$</span>.</p>
  
  <p>Furthermore, the <em>Bellman optimality equation</em> for <span class="math-container">$V^*$</span> (the optimal value function) can be expressed as the Bellman optimality equation for <span class="math-container">$Q^{\pi^*}$</span> (the optimal state-action value function associated with the optimal policy <span class="math-container">$\pi^*$</span>) as follows</p>
  
  <p><span class="math-container">$$
  V^*(s) = \max_{a \in \mathcal{A}(s)} Q^{\pi^*}(s, a)
  $$</span></p>
  
  <p>This is actually shown (or proved) at page 76 of the book "Reinforcement Learning: An Introduction" (1st edition) by Andrew Barto and Richard S. Sutton.</p>
  
  <p>Are there any other functions, apart from the <span class="math-container">$V$</span> and <span class="math-container">$Q$</span> functions defined above, in the RL context? If so, how are they related?  </p>
  
  <p>For example, I've heard of the "advantage" or "continuation" functions. How are these functions related to the <span class="math-container">$V$</span> and <span class="math-container">$Q$</span> functions? When should one be used as opposed to the other? Note that I'm <em>not just</em> asking about the "advantage" or "continuation" functions, but, if possible, any existing function that is used in RL that is similar (in purpose) to these mentioned functions, and how they are related to each other. </p>
  
Tags: <reinforcement-learning><comparison><reference-request><value-functions><bellman-equations>
Title: Apart from the state and state-action value functions, what are other examples of value functions used in RL?
UserDisplayName: nbro
ThreadText: # Apart from the state and state-action value functions, what are other examples of value functions used in RL?
  
  (Asked by: username_0 on 2019-02-14)
  
  In reinforcement learning, we often define two functions, the *state-value function*
  
  $$V^\pi(s) = \mathbb{E}\_{\pi} \left[\sum\_{k=0}^{\infty}
  \gamma^{k}R\_{t+k+1} \Bigg| S\_t=s \right]$$
  
  and the *state-action-value function*
  
  $$Q^\pi(s,a) = \mathbb{E}\_{\pi}\left[\sum\_{k=0}^{\infty} \gamma^{k}R\_{t+k+1}\Bigg|S\_t=s, A\_t=a \right]$$
  
  where $\mathbb{E}\_{\pi}$ means that these functions are defined as the [expectation](https://en.wikipedia.org/wiki/Expected_value) with respect to a fixed policy $\pi$ of what is often called the *return*, $\sum\_{k=0}^{\infty} \gamma^{k}R\_{t+k+1}$, where $\gamma$ is a *discount factor* and $R\_{t+k+1}$ is the *reward* received from the environment (while the agent interacts with it) from time $t$ onwards.
  
  So, both the $V$ and $Q$ functions are defined as expectations of the return (or the *cumulative future discounted reward*), but these expectations have different "conditions" (or are conditioned on different variables). The $V$ function is the expectation (with respect to a fixed policy $\pi$) of the return given that the current state (the state at time $t$) is $s$. The $Q$ function is the expectation (with respect to a fixed policy $\pi$) of the return conditioned on the fact that the current state the agent is in is $s$ and the action the agent takes at $s$ is $a$.
  
  Furthermore, the *Bellman optimality equation* for $V^\*$ (the optimal value function) can be expressed as the Bellman optimality equation for $Q^{\pi^\*}$ (the optimal state-action value function associated with the optimal policy $\pi^\*$) as follows
  
  $$
  V^\*(s) = \max\_{a \in \mathcal{A}(s)} Q^{\pi^\*}(s, a)
  $$
  
  This is actually shown (or proved) at page 76 of the book "Reinforcement Learning: An Introduction" (1st edition) by Andrew Barto and Richard S. Sutton.
  
  Are there any other functions, apart from the $V$ and $Q$ functions defined above, in the RL context? If so, how are they related?
  
  For example, I've heard of the "advantage" or "continuation" functions. How are these functions related to the $V$ and $Q$ functions? When should one be used as opposed to the other? Note that I'm *not just* asking about the "advantage" or "continuation" functions, but, if possible, any existing function that is used in RL that is similar (in purpose) to these mentioned functions, and how they are related to each other.
  
  **Tags:** reinforcement-learning, comparison, reference-request, value-functions, bellman-equations
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-03-07. Score: 6 | Accepted answer
  
  **Advantage function**: $A(s,a) = Q(s,a) - V(s)$
  
  More interesting is the **General Value Function (GVF)**, the expected sum of the (discounted) future values of some arbitrary signal, not necessarily reward. It is therefore a generalization of value function $V(s)$. The GVF is defined on page 459 of the 2nd edition of [Sutton and Barto's RL book](http://incompleteideas.net/book/the-book-2nd.html) as
  $$v\_{\pi,\gamma,C}(s) =\mathbb{E}\left[\left.\sum\_{k=t}^\infty\left(\prod\_{i=t+1}^k \gamma(S\_i)\right)C\_{k+1}\right\rvert S\_t=s, A\_{t:\infty}\sim\pi\right]$$
  where $C\_t \in \mathbb{R}$ is the signal being summed over time.
  
  $\gamma(S\_t)$ is a function $\gamma: \cal{S}\to[0,1]$ allowing the discount rate to depend upon the state. Sutton and Barto call it the termination function. [Some](http://reinforcement-learning.ml/papers/pgmrl2018_sherstan.pdf) call it the continuation function.
  
  Also of note are the **differential value functions**. These are used in the continuing, undiscounted setting. Because there is no discounting, the expected sum of future rewards is unbounded. Instead, we optimize the expected differential reward $R\_{t+1}-r(\pi)$, where $r(\pi)$ is the average reward under policy $\pi$.
  
  $$v\_{\pi,\,diff}(s) = \sum\_a \pi(a|s) \sum\_{s',r} p(s',r|s,a)\left[r-r(\pi)+ v\_{\pi,\,diff}(s')\right]$$
  $$v\_{\*,\,diff}(s) = \max\_a \sum\_{s',r} p(s',r|s,a)\left[r-\max\_\pi r(\pi)+ v\_{\*,\,diff}(s')\right]$$
  
  The differential value functions assume that a single fixed value of $r(\pi)$ exists. That is, they assume the MDP is "ergodic." See section 10.3 of Sutton and Barto for details.
  
  > **Comments:**
  > **username_2**: Regarding the "Some" link: This site [http://reinforcement-learning.ml](http://reinforcement-learning.ml), I assume a previous github pages size, is offline.  
  
  ---
  
  ### Answer by username_0 on 2020-10-29. Score: 0
  
  There is also the simpler **action value function** $$q\_\*(a) = \mathbb{E} \left[ R\_t
  \mid A\_t = a\right],$$ which we try to approximate when solving *context-free* **bandit problems**. You can also similarly define the action value function for *contextual* bandit problems by also conditioning on the context (rather than just on the action).
  
  See chapter 2 of the book [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf#page=48) (2nd edition) by Barto and Sutton for more details.
  
  There are also the **afterstate value functions**, $v(s')$. Check [this question](https://ai.stackexchange.com/q/24816/2444) and [section 6.8](http://incompleteideas.net/book/RLbook2020.pdf#page=158) of the just cited book.
  
  Moreover, there's the **state-action-goal value function**, $q(s, a, g)$, as described in the paper [Hindsight Experience Replay](https://papers.nips.cc/paper/2017/file/453fadbd8a1a3af50a9df4df899537b5-Paper.pdf) (2017), although I am not sure how we can mathematically define it (i.e. as a function of other value functions or as a Bellman equation).
  
  
  ---
  
  