Answers.Body: <p>I think it is pedagogically useful to distinguish between the theory (equations) and the practice (algorithms).</p>
  <p>If you're talking about the definition of the value function (the theory)</p>
  <p><span class="math-container">\begin{align}
  v_{\pi}(s) 
  &amp; \dot{=} \mathbb{E}_{\pi} \left[ G_t \mid S_t = s \right]\\
  &amp;= \mathbb{E}_{\pi} \left[ \sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigl\vert S_t = s \right]\\
  \end{align}</span></p>
  <p>for all <span class="math-container">$s \in \mathcal{S}$</span>, where <span class="math-container">$\dot{=}$</span> means &quot;is defined as&quot; and <span class="math-container">$\mathcal{S}$</span> is the state space, then the value function can be defined in terms of the reward, as it can be clearly seen above. (Note that <span class="math-container">$R_{t+k+1}$</span>, <span class="math-container">$G$</span> and <span class="math-container">$S_t$</span> are random variables, and, in fact, expectations are taken with respect to random variables).</p>
  <p>The definition above can actually be expanded to be a Bellman equation (i.e. a recursive equation) defined in terms of the reward function <span class="math-container">$R(s, a)$</span> of the underlying MDP. However, often, rather than the notation <span class="math-container">$R(s, a)$</span>, you will see <span class="math-container">$p(s', r \mid s, a)$</span> (which represents the combination of the <em>transition probability function</em> and the <em>reward function</em>). Consequently, <strong>the value is a function of the reward</strong>.</p>
  <p>If you're estimating a value function (the practice), e.g. using Q-learning, you don't necessarily use the reward function of the Markov decision process. You can <strong>estimate</strong> the value function by just observing the rewards that you receive while exploring the environment, without really knowing the reward function. But, by exploring the environment, you can actually estimate the reward function. For example, if every time you're in state <span class="math-container">$s$</span> you take action <span class="math-container">$a$</span> and you receive reward <span class="math-container">$r$</span>, then you already know something about the actual underlying reward function. If you explore enough the MDP, you could potentially learn the reward function too (unless it keeps on changing, in that case, it may be more difficult to learn it).</p>
  <p>To conclude, yes, value functions are certainly very related to reward functions and rewards, in ways that you immediately see from the equations that define the value functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T23:17:07.403Z
Answers.LastActivityDate: 2020-10-08T13:00:08.610Z
Answers.LastEditDate: 2020-10-08T13:00:08.610Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Sutton and Barto give a great description in their book <a href="http://www.incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Reinforcement Learning</a>:</p>
  <blockquote>
  <p>Whereas the reward signal indicates what is good in an immediate sense, a value
  function specifies what is good in the long run. Roughly speaking, the value of a state is
  the total amount of reward an agent can expect to accumulate over the future, starting
  from that state. Whereas rewards determine the immediate, intrinsic desirability of
  environmental states, values indicate the long-term desirability of states after taking into
  account the states that are likely to follow and the rewards available in those states. For
  example, a state might always yield a low immediate reward but still have a high value
  because it is regularly followed by other states that yield high rewards. Or the reverse
  could be true. To make a human analogy, rewards are somewhat like pleasure (if high)
  and pain (if low), whereas values correspond to a more refined and farsighted judgment
  of how pleased or displeased we are that our environment is in a particular state.</p>
  </blockquote>
  <p>Mathematically, the value function is the expected sum of discounted rewards from a given state or for a particular action in a given state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-25T23:22:14.013Z
Answers.LastActivityDate: 2022-11-25T23:22:14.013Z
Answers.UserDisplayName: Burrito
Body: <p>To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses to 'generate' those rewards for it to use in the calculation of how 'good' it is to be in the state?</p>
  
Tags: <reinforcement-learning><comparison><value-functions><reward-functions>
Title: What is the relationship between the reward function and the value function?
UserDisplayName: mason7663
ThreadText: # What is the relationship between the reward function and the value function?
  
  (Asked by: username_0 on 2020-04-03)
  
  To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses to 'generate' those rewards for it to use in the calculation of how 'good' it is to be in the state?
  
  **Tags:** reinforcement-learning, comparison, value-functions, reward-functions
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-03. Score: 1 | Accepted answer
  
  I think it is pedagogically useful to distinguish between the theory (equations) and the practice (algorithms).
  
  If you're talking about the definition of the value function (the theory)
  
  \begin{align}
  v\_{\pi}(s)
  & \dot{=} \mathbb{E}\_{\pi} \left[ G\_t \mid S\_t = s \right]\\
  &= \mathbb{E}\_{\pi} \left[ \sum\_{k=0}^\infty \gamma^k R\_{t+k+1} \bigl\vert S\_t = s \right]\\
  \end{align}
  
  for all $s \in \mathcal{S}$, where $\dot{=}$ means "is defined as" and $\mathcal{S}$ is the state space, then the value function can be defined in terms of the reward, as it can be clearly seen above. (Note that $R\_{t+k+1}$, $G$ and $S\_t$ are random variables, and, in fact, expectations are taken with respect to random variables).
  
  The definition above can actually be expanded to be a Bellman equation (i.e. a recursive equation) defined in terms of the reward function $R(s, a)$ of the underlying MDP. However, often, rather than the notation $R(s, a)$, you will see $p(s', r \mid s, a)$ (which represents the combination of the *transition probability function* and the *reward function*). Consequently, **the value is a function of the reward**.
  
  If you're estimating a value function (the practice), e.g. using Q-learning, you don't necessarily use the reward function of the Markov decision process. You can **estimate** the value function by just observing the rewards that you receive while exploring the environment, without really knowing the reward function. But, by exploring the environment, you can actually estimate the reward function. For example, if every time you're in state $s$ you take action $a$ and you receive reward $r$, then you already know something about the actual underlying reward function. If you explore enough the MDP, you could potentially learn the reward function too (unless it keeps on changing, in that case, it may be more difficult to learn it).
  
  To conclude, yes, value functions are certainly very related to reward functions and rewards, in ways that you immediately see from the equations that define the value functions.
  
  
  ---
  
  ### Answer by username_2 on 2022-11-25. Score: 1
  
  Sutton and Barto give a great description in their book [Reinforcement Learning](http://www.incompleteideas.net/book/the-book-2nd.html):
  
  > Whereas the reward signal indicates what is good in an immediate sense, a value
  > function specifies what is good in the long run. Roughly speaking, the value of a state is
  > the total amount of reward an agent can expect to accumulate over the future, starting
  > from that state. Whereas rewards determine the immediate, intrinsic desirability of
  > environmental states, values indicate the long-term desirability of states after taking into
  > account the states that are likely to follow and the rewards available in those states. For
  > example, a state might always yield a low immediate reward but still have a high value
  > because it is regularly followed by other states that yield high rewards. Or the reverse
  > could be true. To make a human analogy, rewards are somewhat like pleasure (if high)
  > and pain (if low), whereas values correspond to a more refined and farsighted judgment
  > of how pleased or displeased we are that our environment is in a particular state.
  
  Mathematically, the value function is the expected sum of discounted rewards from a given state or for a particular action in a given state.
  
  
  ---
  
  