Answers.Body: <blockquote>
  <p>What is the difference between a reward and a value for a given state?</p>
  </blockquote>
  <p>Let us say that an agent took an action from state <span class="math-container">$A$</span> and reached state <span class="math-container">$B$</span> and got a score <span class="math-container">$R$</span>. This instantaneous score the agent received on reaching state <span class="math-container">$B$</span> is called the <em>reward</em>.</p>
  <p>Now, let me introduce you to the concept of <strong>return</strong>. Assume that an agent followed a particular trajectory:</p>
  <pre><code>1. State 1 -&gt; Action 1
  2. Reward 1, State 2 -&gt; Action 2
  3. Reward 2, State 3 -&gt; Action 3
  ...
  n. Reward n-1, State n (Terminated)
  </code></pre>
  <p>Return (often denoted by <span class="math-container">$G$</span>) is the sum total of all the rewards obtained by starting from a state State 1 and following a policy.</p>
  <p>So, the definition of the <strong>return</strong> is</p>
  <p><span class="math-container">$$G(s_1) = R_1 + R_2 + R_3 + ... = \sum_{i=1}^{\infty}R_i$$</span></p>
  <p>Sometimes (most often) these sequences never terminate, so we include a discount factor (Greek letter gamma, <span class="math-container">$\gamma$</span>) to rewards obtained in the future.</p>
  <p>The definition of the discounted return <span class="math-container">$G$</span> is</p>
  <p><span class="math-container">$$G(s_1) = R_1 + \gamma R_2 + \gamma^2 R_3 + ... = \sum_{i=1}^{\infty}\gamma^{i-1} R_i $$</span></p>
  <p><span class="math-container">$\gamma$</span> is a number between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>: it defines how much importance the agent gives to long-term rewards. For a smaller value of <span class="math-container">$\gamma$</span>, more importance is given for short-term rewards.</p>
  <p>Now, coming back to your question. A value of the state is the <strong>expected return</strong> for an agent starting from that state and following a particular policy. In the case of stochastic policies (policies that have inherent randomness) and/or for environments with stochastic transition probabilities and/or stochastic rewards, the value is the sum of (the returns of all trajectories multiplied by the probability of taking that trajectory).</p>
  <blockquote>
  <p>And should the initial values of the states always be set equal to zero?</p>
  </blockquote>
  <p>Not necessary, zero initialization is one of many ways to initialize. Random initialization is another method. It depends on the environment setting.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-19T11:19:19.333Z
Answers.LastActivityDate: 2021-06-21T10:33:26.947Z
Answers.LastEditDate: 2021-06-21T10:33:26.947Z
Answers.UserDisplayName: Sooryakiran Pallikulathil
Answers.Body: <p>Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step <span class="math-container">$s, a, r, s'$</span> for state, action, reward, next state, then the reward <span class="math-container">$r$</span> is allowed to depend on all three of <span class="math-container">$s, a, s'$</span>, and it can also be from a random distribution of real numbers, not just a single number.</p>
  <p>It is however OK to associate a single number reward with each state, for either leaving that state (when it is <span class="math-container">$s$</span> or <span class="math-container">$s_t$</span> in the sequence) or arriving in it (when it is <span class="math-container">$s'$</span> or <span class="math-container">$s_{t+1}$</span>). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.</p>
  <p>State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:</p>
  <p><span class="math-container">$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$</span></p>
  <p>In English: &quot;The expected discounted sum of all future rewards when starting from a given state and following a specific policy.&quot; The discounted sum is usually called the return or the utility associated with the state.</p>
  <blockquote>
  <p>What is the difference between a reward and a value for a given state?</p>
  </blockquote>
  <p>A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to make decisions.</p>
  <blockquote>
  <p>And should the initial values of the states always be set equal to zero?</p>
  </blockquote>
  <p>Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:</p>
  <ul>
  <li><p>Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improve speed of convergence depending on how good the guesses are.</p>
  </li>
  <li><p>Random values - this may happen if you use a neural network.</p>
  </li>
  <li><p>Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-19T11:19:33.840Z
Answers.LastActivityDate: 2021-06-20T08:55:08.730Z
Answers.LastEditDate: 2021-06-20T08:55:08.730Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I encountered the situation in which I had to set the rewards for the agent, but looking at the theory, I have found that each state has also a value, which is found using the value iteration.</p>
  <p>So, my doubt is: <strong>What is the difference between a reward and a value for a given state? And should the initial values of the states always be set equal to zero?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-19T12:09:31.250Z
Comments.Text: [Here](https://ai.stackexchange.com/q/21628/2444) and [here](https://ai.stackexchange.com/q/13135/2444) are two related questions.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><rewards><value-iteration><return>
Title: What is the difference between a reward and a value for a given state?
UserDisplayName: dcr
ThreadText: # What is the difference between a reward and a value for a given state?
  
  (Asked by: username_0 on 2021-06-19)
  
  I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I encountered the situation in which I had to set the rewards for the agent, but looking at the theory, I have found that each state has also a value, which is found using the value iteration.
  
  So, my doubt is: **What is the difference between a reward and a value for a given state? And should the initial values of the states always be set equal to zero?**
  
  **Tags:** reinforcement-learning, comparison, rewards, value-iteration, return
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/21628/2444) and [here](https://ai.stackexchange.com/q/13135/2444) are two related questions.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-06-19. Score: 4 | Accepted answer
  
  Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be from a random distribution of real numbers, not just a single number.
  
  It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s\_t$ in the sequence) or arriving in it (when it is $s'$ or $s\_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.
  
  State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:
  
  $$v\_{\pi}(s) = \mathbb{E}\_{\pi}[\sum\_{k=0}^{\infty} \gamma^k R\_{t+k+1} | S\_t=s]$$
  
  In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.
  
  > What is the difference between a reward and a value for a given state?
  
  A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to make decisions.
  
  > And should the initial values of the states always be set equal to zero?
  
  Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:
  
  * Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improve speed of convergence depending on how good the guesses are.
  * Random values - this may happen if you use a neural network.
  * Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.
  
  
  ---
  
  ### Answer by username_2 on 2021-06-19. Score: 3
  
  > What is the difference between a reward and a value for a given state?
  
  Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the agent received on reaching state $B$ is called the *reward*.
  
  Now, let me introduce you to the concept of **return**. Assume that an agent followed a particular trajectory:
  
  ```
  1. State 1 -> Action 1
  2. Reward 1, State 2 -> Action 2
  3. Reward 2, State 3 -> Action 3
  ...
  n. Reward n-1, State n (Terminated)
  
  ```
  
  Return (often denoted by $G$) is the sum total of all the rewards obtained by starting from a state State 1 and following a policy.
  
  So, the definition of the **return** is
  
  $$G(s\_1) = R\_1 + R\_2 + R\_3 + ... = \sum\_{i=1}^{\infty}R\_i$$
  
  Sometimes (most often) these sequences never terminate, so we include a discount factor (Greek letter gamma, $\gamma$) to rewards obtained in the future.
  
  The definition of the discounted return $G$ is
  
  $$G(s\_1) = R\_1 + \gamma R\_2 + \gamma^2 R\_3 + ... = \sum\_{i=1}^{\infty}\gamma^{i-1} R\_i $$
  
  $\gamma$ is a number between $0$ and $1$: it defines how much importance the agent gives to long-term rewards. For a smaller value of $\gamma$, more importance is given for short-term rewards.
  
  Now, coming back to your question. A value of the state is the **expected return** for an agent starting from that state and following a particular policy. In the case of stochastic policies (policies that have inherent randomness) and/or for environments with stochastic transition probabilities and/or stochastic rewards, the value is the sum of (the returns of all trajectories multiplied by the probability of taking that trajectory).
  
  > And should the initial values of the states always be set equal to zero?
  
  Not necessary, zero initialization is one of many ways to initialize. Random initialization is another method. It depends on the environment setting.
  
  
  ---
  
  