Answers.Body: <blockquote>
  <p>I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number 3 or to continue to the state with a number 4 to collect the more reward?</p>
  </blockquote>
  <p>Which is better is determined by looking at the expected return from either choice, with higher expected returns being better.</p>
  <p>The return from travelling one time step to the &quot;4 on exit&quot; state, is 2 as you have shown, due to discounting. That is assuming the 4 is gained by taking a separate &quot;exit&quot; action once in that position. That is, there is no combined &quot;move and exit&quot; action that only takes one time step - whether or not such actions exist in this environment makes a large change to your example and what will be optimal, so it is really important to be clear about that.</p>
  <p>The return from exiting immediately in the &quot;3 on exit&quot; state is 3.</p>
  <p>3 is larger than 2, so if the agent finds itself in the &quot;3 on exit&quot; state, then it should exit immediately to get the best expected return.</p>
  <p>Possibly what might be making this harder to understand is the role that discounting takes. Sometimes it appears to be used to &quot;fix&quot; infinite rewards from continuing environments. However, discounting is part of the definition of return, and it changes what counts as optimal. With a low discount factor, such as <span class="math-container">$0.5$</span>, then it can be optimal to take a lower reward sooner as opposed to a larger reward later. The value of the discount factor allows to make that comparison exactly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-16T09:05:13.107Z
Answers.Text: Thanks for the reply! From a numbered square, if the agent wants they can exit to a terminal state X which will give them a reward equal to the number in the square. You are right, there is no combined move and exit. So we can either move or exit. Given this, we would then exit at state 3? The state before state 3 will then become $(3+0.5(3))?$ Where $R = 3, \gamma=0.5 , V' = 3$
Answers.UserDisplayName: Krellex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-16T09:07:50.157Z
Answers.Text: @Krellex You already have the value of the state *below* the "3 on exit" state correct. An agent in that state acting optimally would travel up, gaining 0 immediate reward and arriving in the state "3 on exit" which has a value of 3. So $V^*(s) = 0 + 0.5(3) = 1.5$. We can tell that is optimal in this simple environment as there is no chain of moves that could possibly gain a higher return than 1.5 from that location (in more complex environments you won't be able to see that so clearly)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-16T09:16:45.600Z
Answers.Text: Thanks so much for this clear explanation, really helped me understand!
Answers.UserDisplayName: Krellex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-16T08:56:30.233Z
Answers.LastActivityDate: 2022-04-16T09:01:45.903Z
Answers.LastEditDate: 2022-04-16T09:01:45.903Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount inside the numbered square or they can choose to not do this and just move to the state grid square.</p>
  <p>Since this is a determinisitc grid, I have utilised the following Bellman equation:
  <span class="math-container">$$V(s) = max_a(R(s,a)+\gamma V(s'))$$</span></p>
  <p>Where <span class="math-container">$\gamma=0.5$</span> and any movement reward is <span class="math-container">$0$</span>, since this will allow the agent to have a balance of thinking long-term and short-term.</p>
  <p>I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number <span class="math-container">$3$</span> or to continue to the state with a number <span class="math-container">$4$</span> to collect the more reward? I have determined and marked (X) the terminal states, where with my current calculations, I feel the agent should exit.</p>
  <p><a href="https://i.sstatic.net/9U1tm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9U1tm.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-17T08:23:55.843Z
Comments.Text: Hi Krellex, I have rolled back the image edit as the new image stops looking line an MDP to me. By losing the arrows and the crosses, there is no indication of the actions. By losing the separate reward numbers, there is no indication of reward, only a list of values. In general, there is no need to "fix" parts of the question when you receive an answer that corrects something in it. You can and should make edits that when someone adds a comment asking you to clarify something, However, it should be avoided when there is an answer which refers to that part of your question.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-17T08:32:27.237Z
Comments.Text: @NeilSlater Ah ok! Thanks for changing it back. I thought my edit might have made it clearer, but you're right, the older one was much clearer. Thanks for the update.
Comments.UserDisplayName: Krellex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-17T08:40:36.107Z
Comments.Text: By the way, your diagram happens to show an entirely self-consistent policy and matching state values. That is, all the values are correct given the policy shown by the arrows and crosses. The only thing "wrong" about it is that the policy is not the optimal one that you were looking for - in just one state. Which is what your question was about, so that's fine.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-17T08:48:04.350Z
Comments.Text: @NeilSlater I have implemented the changes for State 3 so it's optimal. I have also updated state (1,3) to exit at state (2,3) to collect a reward of 2 since this would be the most optimal movement since it returns the highest value from all actions given a discount value of $\gamma=0.5$ I think that would be correct for that state also?
Comments.UserDisplayName: Krellex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-17T09:03:21.967Z
Comments.Text: For state at (2,3) with exit value of 2, you have a choice. Both moving up towards (4,3) for an eventual reward of 8 and taking the immediate exit for reward of 2 have a value of 2. In ties for maximum return, then *both* actions are optimal. I suspect the grid world has been deliberately set up to show that as a discussion point.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-17T09:15:37.000Z
Comments.Text: @NeilSlater Ah ok! That make sense. Since they're equal, we can choose either one? I have decided to go with the approach shown in the image above for that state since this will give the agent a larger reward.
Comments.UserDisplayName: Krellex
Tags: <reinforcement-learning><deep-rl><bellman-equations><deterministic-policy><stopping-conditions>
Title: Determining to terminate at a reward or not
UserDisplayName: Krellex
ThreadText: # Determining to terminate at a reward or not
  
  (Asked by: username_0 on 2022-04-16)
  
  I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount inside the numbered square or they can choose to not do this and just move to the state grid square.
  
  Since this is a determinisitc grid, I have utilised the following Bellman equation:
  $$V(s) = max\_a(R(s,a)+\gamma V(s'))$$
  
  Where $\gamma=0.5$ and any movement reward is $0$, since this will allow the agent to have a balance of thinking long-term and short-term.
  
  I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number $3$ or to continue to the state with a number $4$ to collect the more reward? I have determined and marked (X) the terminal states, where with my current calculations, I feel the agent should exit.
  
  [![enter image description here](https://i.sstatic.net/9U1tm.png)](https://i.sstatic.net/9U1tm.png)
  
  **Tags:** reinforcement-learning, deep-rl, bellman-equations, deterministic-policy, stopping-conditions
  
  > **Comments:**
  > **username_1**: Hi username_0, I have rolled back the image edit as the new image stops looking line an MDP to me. By losing the arrows and the crosses, there is no indication of the actions. By losing the separate reward numbers, there is no indication of reward, only a list of values. In general, there is no need to "fix" parts of the question when you receive an answer that corrects something in it. You can and should make edits that when someone adds a comment asking you to clarify something, However, it should be avoided when there is an answer which refers to that part of your question.  
  > **username_0**: @username_1 Ah ok! Thanks for changing it back. I thought my edit might have made it clearer, but you're right, the older one was much clearer. Thanks for the update.  
  > **username_1**: By the way, your diagram happens to show an entirely self-consistent policy and matching state values. That is, all the values are correct given the policy shown by the arrows and crosses. The only thing "wrong" about it is that the policy is not the optimal one that you were looking for - in just one state. Which is what your question was about, so that's fine.  
  > **username_0**: @username_1 I have implemented the changes for State 3 so it's optimal. I have also updated state (1,3) to exit at state (2,3) to collect a reward of 2 since this would be the most optimal movement since it returns the highest value from all actions given a discount value of $\gamma=0.5$ I think that would be correct for that state also?  
  > **username_1**: For state at (2,3) with exit value of 2, you have a choice. Both moving up towards (4,3) for an eventual reward of 8 and taking the immediate exit for reward of 2 have a value of 2. In ties for maximum return, then *both* actions are optimal. I suspect the grid world has been deliberately set up to show that as a discussion point.  
  > **username_0**: @username_1 Ah ok! That make sense. Since they're equal, we can choose either one? I have decided to go with the approach shown in the image above for that state since this will give the agent a larger reward.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-16. Score: 3 | Accepted answer
  
  > I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number 3 or to continue to the state with a number 4 to collect the more reward?
  
  Which is better is determined by looking at the expected return from either choice, with higher expected returns being better.
  
  The return from travelling one time step to the "4 on exit" state, is 2 as you have shown, due to discounting. That is assuming the 4 is gained by taking a separate "exit" action once in that position. That is, there is no combined "move and exit" action that only takes one time step - whether or not such actions exist in this environment makes a large change to your example and what will be optimal, so it is really important to be clear about that.
  
  The return from exiting immediately in the "3 on exit" state is 3.
  
  3 is larger than 2, so if the agent finds itself in the "3 on exit" state, then it should exit immediately to get the best expected return.
  
  Possibly what might be making this harder to understand is the role that discounting takes. Sometimes it appears to be used to "fix" infinite rewards from continuing environments. However, discounting is part of the definition of return, and it changes what counts as optimal. With a low discount factor, such as $0.5$, then it can be optimal to take a lower reward sooner as opposed to a larger reward later. The value of the discount factor allows to make that comparison exactly.
  
  > **Comments:**
  > **username_0**: Thanks for the reply! From a numbered square, if the agent wants they can exit to a terminal state X which will give them a reward equal to the number in the square. You are right, there is no combined move and exit. So we can either move or exit. Given this, we would then exit at state 3? The state before state 3 will then become $(3+0.5(3))?$ Where $R = 3, \gamma=0.5 , V' = 3$  
  > **username_1**: @username_0 You already have the value of the state *below* the "3 on exit" state correct. An agent in that state acting optimally would travel up, gaining 0 immediate reward and arriving in the state "3 on exit" which has a value of 3. So $V^*(s) = 0 + 0.5(3) = 1.5$. We can tell that is optimal in this simple environment as there is no chain of moves that could possibly gain a higher return than 1.5 from that location (in more complex environments you won't be able to see that so clearly)  
  > **username_0**: Thanks so much for this clear explanation, really helped me understand!  
  
  ---
  
  