Answers.Body: <p>In reinforcement learning, you do not usually wish to train an agent to learn to take actions for maximising the highest possible reward. Instead you wish to train the agent to take actions that maximise the <em>expected</em> reward. An optimal agent is one that, on average, gains the most reward. It is not one that relies on being lucky to score the highest possible reward.</p>
  <p>As such, any reward experienced is assumed to be a fair sample of one of the possible rewards received when taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. Provided there is no hidden state, and thus the Markov property holds, and the agent samples <span class="math-container">$s, a$</span> enough times to determine the expected reward from that combination, then this means the agent's estimates will converge towards the true expected action values.</p>
  <p>As a simple example, if the reward from taking action <span class="math-container">$a_1$</span> in state <span class="math-container">$s_1$</span> was <span class="math-container">$-1$</span> most of the time, but 5% of the time it was <span class="math-container">$+10$</span>, and the reward from taking action <span class="math-container">$a_2$</span> in the same state was always <span class="math-container">$+1$</span>, and both action choices transitioned to state <span class="math-container">$s_2$</span>, then action <span class="math-container">$a_2$</span> should be considered a more optimal choice than action <span class="math-container">$a_1$</span>. Despite the fact that it is <em>possible</em> to get a higher reward in <span class="math-container">$a_1$</span>, the higher expected reward from <span class="math-container">$a_2$</span> is a better choice.</p>
  <p>If the agent is allowed to know the true expected immediated reward from <span class="math-container">$s, a$</span>, the you could use that instead of using the sampled values. You should not use the maximum observed reward so far to update the agent, because that is a biased measure of the expected reward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-02T22:20:09.253Z
Answers.Text: lets say you are in state s and you take action a1 and you get -1 or you can stake action a2 from state s  and get +10. Lets say you save the expirience where you take action a1 and get -1 from state s and you save that to the  experience buffer.  When you got to train the network you get that example from the expirience buffer and input it into the networks and bellman equation. when you calculate the loss and adjust the weights wont that prompt the network to take action a1 more even though it is not the best option in that situation ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T07:19:26.590Z
Answers.Text: @Stef In your example, I would expect the network to reduce its estimate of expected return from taking $a_1$ in state $s$. The greedy action at that point is more likely to be $a_2$. However, the whole process is stochastic, and it is possible that the initial estimates for $a_1$ are higher than those for $a_2$ for a few iterations. Once the nwtork has been updated towards targets for $\hat{q}(s,a_1) = -1$ and $\hat{q}(s,a_2) = 10$ after taking several sample of them from memory, then the estimate for taking action $a_2$ will be higher and the agent will prefer it.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T07:21:11.237Z
Answers.Text: @Stef From your comment, I wonder if you are missing the detail of what the difference between a state value and an action value is? You seem to think learning about the result from action $a_1$ should heavily influence the estimate of the result from $a_2$? In tabular Q learning or SARSA those estimates are entirely separate, and in approximation-based systems (e.g. neural nets), they will still have at least some separate learnable weight parameters where the agent can easily learn the difference
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T15:37:30.150Z
Answers.Text: so when the network learns from experience (s,a,r,s') and you train you model on the experience that gives you -1 as the reward. During training the model is just trying to predict the actual reward of that s a r s' not the best possible action in that state? Does the model take the same action a as in the experience (s,a,r,s') and the try to adjust weights to yield reward r ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T15:52:25.870Z
Answers.Text: @Stef Yes Q-learning does not learn which action to take directly, but is a value-based method that learns what the expected future sum of returns is for each action in a state, and then checks those values to decide which action to take. I don't understand your second question, you may need to re-phrase it as a full new question, so we can figure out what you mean, if you think it is important to your understanding. Main issue is that the word "model" means something in RL, and there is no model in Q learning, so you are calling *something* the model, but I don't know what.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T16:23:14.317Z
Answers.Text: With model I am referring to the Q network in DQN. I am asking when you take experience (s,a,r,s') that has r as -1 when you use that experience to train the Q network when you input state s into the Q network does the Q network take the same action as in experience (s,a,r,s')and the adjust its estimate of reward r to be closer to -1 (I understand it is also taking into account the target network estimate as well). Or does it take whatever action it thinks is best not necessarily  the same action as in (s,a,r,s') ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T17:53:07.820Z
Answers.Text: @Stef The Q network doesn't "take actions" at all. There is the concept of the *greedy action* which is the action that Q, the action value function, predicts will have the best return. The network doesn't choose that though - other code must do that. The Q network also does not approximate or estimate $r$, and will not "adjust its estimate of reward r to be closer to -1" - the part about the target network estimate is not some optional extra unless you are working with a discount factor of $0$. The Q network estimates the *action value* of taking an action.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T18:20:01.730Z
Answers.Text: So when inputing (s,a1,r,s') for training, the q network updates its parameters to estimate the action value of (s,a1) instead of (s,a2) even though s,a2 is the action with the highest value. In this example the network is focused on getting a better action value estimate when you take action 1 from state s even if it doesn't necessarily lead to the highest action value its just trying to get a correct estimate of what the value is. Is that understanding correct ?
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T18:30:21.483Z
Answers.Text: @Stef It updates the estimate action value for the action that was taken. This is the same in principle as any supervised learning update. The approximation updates the estimate of the output (q) associated with a given input (s,a). To make things a *little* more complicated, the architecture of the network in DQN might present all actions at once and need some special handling - whether that is important to you depends on whether you are implementing a DQN agent, or using a DQN agent library - the library version will do the necessary handling to only update the specific action being updated.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T18:45:39.760Z
Answers.Text: Perfect I finally understand DQN I really appreciate it thanks for taking the time to help me ! I can't wait to start building !
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-02T19:19:28.963Z
Answers.LastActivityDate: 2023-07-02T19:19:28.963Z
Answers.UserDisplayName: Neil Slater
Body: <p>My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not the highest possible reward you can get in that state so then when you are calculating the loss and updating the weights isn't it training the network to pick a reward that isn't the maximum possible reward in that state. I realize it is also taking the target estimate into account as well when adjust the weights buy I am talking about specifically the r.</p>
  
Tags: <neural-networks><reinforcement-learning><deep-learning><deep-rl><dqn>
Title: When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?
UserDisplayName: Stef
ThreadText: # When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?
  
  (Asked by: username_0 on 2023-07-02)
  
  My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not the highest possible reward you can get in that state so then when you are calculating the loss and updating the weights isn't it training the network to pick a reward that isn't the maximum possible reward in that state. I realize it is also taking the target estimate into account as well when adjust the weights buy I am talking about specifically the r.
  
  **Tags:** neural-networks, reinforcement-learning, deep-learning, deep-rl, dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-02. Score: 3 | Accepted answer
  
  In reinforcement learning, you do not usually wish to train an agent to learn to take actions for maximising the highest possible reward. Instead you wish to train the agent to take actions that maximise the *expected* reward. An optimal agent is one that, on average, gains the most reward. It is not one that relies on being lucky to score the highest possible reward.
  
  As such, any reward experienced is assumed to be a fair sample of one of the possible rewards received when taking action $a$ in state $s$. Provided there is no hidden state, and thus the Markov property holds, and the agent samples $s, a$ enough times to determine the expected reward from that combination, then this means the agent's estimates will converge towards the true expected action values.
  
  As a simple example, if the reward from taking action $a\_1$ in state $s\_1$ was $-1$ most of the time, but 5% of the time it was $+10$, and the reward from taking action $a\_2$ in the same state was always $+1$, and both action choices transitioned to state $s\_2$, then action $a\_2$ should be considered a more optimal choice than action $a\_1$. Despite the fact that it is *possible* to get a higher reward in $a\_1$, the higher expected reward from $a\_2$ is a better choice.
  
  If the agent is allowed to know the true expected immediated reward from $s, a$, the you could use that instead of using the sampled values. You should not use the maximum observed reward so far to update the agent, because that is a biased measure of the expected reward.
  
  > **Comments:**
  > **username_0**: lets say you are in state s and you take action a1 and you get -1 or you can stake action a2 from state s  and get +10. Lets say you save the expirience where you take action a1 and get -1 from state s and you save that to the  experience buffer.  When you got to train the network you get that example from the expirience buffer and input it into the networks and bellman equation. when you calculate the loss and adjust the weights wont that prompt the network to take action a1 more even though it is not the best option in that situation ?  
  > **username_1**: @username_0 In your example, I would expect the network to reduce its estimate of expected return from taking $a_1$ in state $s$. The greedy action at that point is more likely to be $a_2$. However, the whole process is stochastic, and it is possible that the initial estimates for $a_1$ are higher than those for $a_2$ for a few iterations. Once the nwtork has been updated towards targets for $\hat{q}(s,a_1) = -1$ and $\hat{q}(s,a_2) = 10$ after taking several sample of them from memory, then the estimate for taking action $a_2$ will be higher and the agent will prefer it.  
  > **username_1**: @username_0 From your comment, I wonder if you are missing the detail of what the difference between a state value and an action value is? You seem to think learning about the result from action $a_1$ should heavily influence the estimate of the result from $a_2$? In tabular Q learning or SARSA those estimates are entirely separate, and in approximation-based systems (e.g. neural nets), they will still have at least some separate learnable weight parameters where the agent can easily learn the difference  
  > **username_0**: so when the network learns from experience (s,a,r,s') and you train you model on the experience that gives you -1 as the reward. During training the model is just trying to predict the actual reward of that s a r s' not the best possible action in that state? Does the model take the same action a as in the experience (s,a,r,s') and the try to adjust weights to yield reward r ?  
  > **username_1**: @username_0 Yes Q-learning does not learn which action to take directly, but is a value-based method that learns what the expected future sum of returns is for each action in a state, and then checks those values to decide which action to take. I don't understand your second question, you may need to re-phrase it as a full new question, so we can figure out what you mean, if you think it is important to your understanding. Main issue is that the word "model" means something in RL, and there is no model in Q learning, so you are calling *something* the model, but I don't know what.  
  > **username_0**: With model I am referring to the Q network in DQN. I am asking when you take experience (s,a,r,s') that has r as -1 when you use that experience to train the Q network when you input state s into the Q network does the Q network take the same action as in experience (s,a,r,s')and the adjust its estimate of reward r to be closer to -1 (I understand it is also taking into account the target network estimate as well). Or does it take whatever action it thinks is best not necessarily  the same action as in (s,a,r,s') ?  
  > **username_1**: @username_0 The Q network doesn't "take actions" at all. There is the concept of the *greedy action* which is the action that Q, the action value function, predicts will have the best return. The network doesn't choose that though - other code must do that. The Q network also does not approximate or estimate $r$, and will not "adjust its estimate of reward r to be closer to -1" - the part about the target network estimate is not some optional extra unless you are working with a discount factor of $0$. The Q network estimates the *action value* of taking an action.  
  > **username_0**: So when inputing (s,a1,r,s') for training, the q network updates its parameters to estimate the action value of (s,a1) instead of (s,a2) even though s,a2 is the action with the highest value. In this example the network is focused on getting a better action value estimate when you take action 1 from state s even if it doesn't necessarily lead to the highest action value its just trying to get a correct estimate of what the value is. Is that understanding correct ?  
  > **username_1**: @username_0 It updates the estimate action value for the action that was taken. This is the same in principle as any supervised learning update. The approximation updates the estimate of the output (q) associated with a given input (s,a). To make things a *little* more complicated, the architecture of the network in DQN might present all actions at once and need some special handling - whether that is important to you depends on whether you are implementing a DQN agent, or using a DQN agent library - the library version will do the necessary handling to only update the specific action being updated.  
  > **username_0**: Perfect I finally understand DQN I really appreciate it thanks for taking the time to help me ! I can't wait to start building !  
  
  ---
  
  