Answers.Body: <blockquote>
  <p>In reinforcement learning, is the value of terminal/goal state always zero?</p>
  </blockquote>
  <p>Yes, always for episodic problems, the value of a terminal state is always zero, from the definition.</p>
  <p>The value of a state <span class="math-container">$v(s)$</span> is the expected sum (perhaps discounted) of rewards from all <em>future</em> time steps. There are no future time steps when in a terminal state, so this sum must be zero.</p>
  <p>For the sake of consistent maths notation, you can consider a terminal state to be &quot;absorbing&quot;, i.e. any transition out of it results in zero reward and returning to the same terminal state. Then you can use the definition of value function to show the same thing:</p>
  <p><span class="math-container">$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_{t} = s]$$</span></p>
  <p>If <span class="math-container">$s = s_T$</span>, the terminal state, then all the &quot;future rewards&quot; from <span class="math-container">$k=0$</span> onwards starting with reward <span class="math-container">$R_{t+1}$</span> must be zero. This is consistent with the reward <span class="math-container">$R_{t}$</span>, i.e. the reward when transitioning <em>to</em> the terminal state, being any value.</p>
  <p>You can show similar using action value functions, if you accept a &quot;null&quot; action in the terminal state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T12:25:51.370Z
Answers.Text: To be clearer, the value of the terminal state is zero **by definition** of the terminal state. In theory, if you keep on receiving reward once you're in the terminal state, then the value won't be zero.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T13:56:00.713Z
Answers.Text: @nbro: If you keep on receiving reward once you're in a "terminal state" then it wasn't a terminal state. A terminal state is where the MDP stops. I think that is the more fundamental definition, the rest is consequences and book-keeping around that. The use of "absorbing" states is an example of that book keeping, and it is used so that some of the theory for continuous and episodic MDPs can use the same notation and derivations.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T10:21:12.137Z
Answers.LastActivityDate: 2020-11-01T15:11:22.057Z
Answers.LastEditDate: 2020-11-01T15:11:22.057Z
Answers.UserDisplayName: Neil Slater
Body: <p>Let's assume we are in a <span class="math-container">$3 \times 3$</span> grid world with states numbered as <span class="math-container">$0,1, \dots, 8$</span>. Suppose that the goal state is <span class="math-container">$8$</span>, the reward of landing in the goal state is <span class="math-container">$10$</span>, and the reward of just wandering around in the grid world is <span class="math-container">$0$</span>. Is the state-value of state <span class="math-container">$8$</span> always <span class="math-container">$0$</span>?</p>
  
Tags: <reinforcement-learning><environment><value-functions>
Title: In reinforcement learning, is the value of terminal/goal state always zero?
UserDisplayName: Bhuwan Bhatt
ThreadText: # In reinforcement learning, is the value of terminal/goal state always zero?
  
  (Asked by: username_0 on 2020-02-07)
  
  Let's assume we are in a $3 \times 3$ grid world with states numbered as $0,1, \dots, 8$. Suppose that the goal state is $8$, the reward of landing in the goal state is $10$, and the reward of just wandering around in the grid world is $0$. Is the state-value of state $8$ always $0$?
  
  **Tags:** reinforcement-learning, environment, value-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-07. Score: 3 | Accepted answer
  
  > In reinforcement learning, is the value of terminal/goal state always zero?
  
  Yes, always for episodic problems, the value of a terminal state is always zero, from the definition.
  
  The value of a state $v(s)$ is the expected sum (perhaps discounted) of rewards from all *future* time steps. There are no future time steps when in a terminal state, so this sum must be zero.
  
  For the sake of consistent maths notation, you can consider a terminal state to be "absorbing", i.e. any transition out of it results in zero reward and returning to the same terminal state. Then you can use the definition of value function to show the same thing:
  
  $$v\_{\pi}(s) = \mathbb{E}\_{\pi}[\sum\_{k=0}^{\infty} \gamma^k R\_{t+k+1} | S\_{t} = s]$$
  
  If $s = s\_T$, the terminal state, then all the "future rewards" from $k=0$ onwards starting with reward $R\_{t+1}$ must be zero. This is consistent with the reward $R\_{t}$, i.e. the reward when transitioning *to* the terminal state, being any value.
  
  You can show similar using action value functions, if you accept a "null" action in the terminal state.
  
  > **Comments:**
  > **username_2**: To be clearer, the value of the terminal state is zero **by definition** of the terminal state. In theory, if you keep on receiving reward once you're in the terminal state, then the value won't be zero.  
  > **username_1**: @nbro: If you keep on receiving reward once you're in a "terminal state" then it wasn't a terminal state. A terminal state is where the MDP stops. I think that is the more fundamental definition, the rest is consequences and book-keeping around that. The use of "absorbing" states is an example of that book keeping, and it is used so that some of the theory for continuous and episodic MDPs can use the same notation and derivations.  
  
  ---
  
  