Answers.Body: <p>Let's try to understand what actually happens when learning. You start with random values for Q(s,a), the function that estimates the reward of an action given a state s, and you then get an immediate reward and then your loss is difference between the given reward + gamma *future estimated reward, i.e. <span class="math-container">$loss = Q(s,a) - (reward + gamma * Q(s', max_a))$</span>.</p>
  <p>Now if gamma=0, it means the loss would not be dependent on what happens afterwards - since the loss would simply be <span class="math-container">$Q(s,a) = reward$</span>. So eventually Q(s,a) would simply converge to be the immediate reward of action a in state s. This means the agent would not end up learning the optimal strategy, since it would choose the action that maximizes the immediate reward without caring what happens afterwards, even if the step after it will die.</p>
  <p>In order to get some intuition on how in practice it actually helps the agent look at the future, try to imagine how this would work out in practice. In the beginning rewards would be random, and at some point it will get to a state before the final episode reward would be very positive if the agent won. Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state - in which case since gamma &gt; 0 it would optimize Q(s,a) to something positive. After millions of s,a pairs you can imagine how this will create a good agent.</p>
  <p>Regarding your other suggestion - I am not an expert but I imagine there might be ways to take into account the final episode reward in a different manner - but the main nice thing about DQN is that it is simple and local - and even though you just update based on samples from local actions, with time the network will (hopefully) converge to be optimal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T20:57:07.090Z
Answers.Text: Thank you for your reply! I had to let this problem go for awhile so I haven't read your answer until now.
  
  It is this part "Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state" that I have a hard time to intuitively understand. I'll try to read it a few more times and think about it.
  
  It just sounds so weird that it can learn something without knowing the state's position in the chain of events. :)
Answers.UserDisplayName: DanneP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-20T06:13:52.977Z
Answers.LastActivityDate: 2023-03-20T06:20:03.050Z
Answers.LastEditDate: 2023-03-20T06:20:03.050Z
Answers.UserDisplayName: dan12345
Body: <p>Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.</p>
  <p><strong>Correct me if I'm wrong (1):</strong><br>
  To train a Deep Q-learning network, every N:th step of action taken, a batch of random samples from replay memory are fed into the model's training function. These samples are totally unrelated to each other, they are unordered and we can't tell if a particular sample is taken from an episode which ended with a positive or negative reward. This is all there is to the training part of a Deep Q-learning network.</p>
  <p>From what I've read, the meaning of the discount factor (gamma) is to decide &quot;how much we value future rewards.&quot;</p>
  <p><strong>Correct me if I'm wrong (2):</strong><br>
  With gamma = 0: The model only cares about the reward from when the state goes from A to B given the action C, and the training is encouraging this behaviour of the model with the reward. <br>
  With gamma = 0.9: The model cares about future rewards and acts based on what might come 5 (or what number we might want) steps from <em>this</em>.</p>
  <p>I know I have gotten something wrong somewhere (or on multiple places :D): How can this small value, gamma, make the model train itself &quot;for the future&quot; when nobody (not even human) kan tell what the reward 5 steps from a particular step will be?</p>
  <p>To me it would have made sense if we had not only 32 random tuples of (state, action, reward, state') but also a &quot;episode end reward&quot; and a &quot;steps until reward&quot; for each tuple and we in some way put these two new pieces of information into the function.</p>
  <p>I'd really like to understand what the gamma does and how. I'm far from a maths person but from the code I've seen in a few tutorials and examples, I can't even begin to understand how this 0.9 or 0.99 can enable the model to train for future reward.</p>
  <p>Please help :)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-19T09:17:12.320Z
Comments.Text: You may need to show some of the equations you are looking at. You refer multiple times to a variable called "sigma", but I am not familiar with a use for a variable labelled with $\sigma$ or $\Sigma$ in the usual equations of reinforcement learning or deep Q learning.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-19T17:18:40.087Z
Comments.Text: Hello again @NeilSlater, I'm sorry, as I wrote I suddenly started to name the factor "sigma" when it should have been "gamma". I've now corrected the original post. 
  
  Again, I'm sorry for the mistake, my only excuse being that I was to tired after trying to fix my neural net all day.
Comments.UserDisplayName: DanneP
Tags: <reinforcement-learning><dqn>
Title: How does the discount factor (gamma) and memory replay work in Deep Q-learning?
UserDisplayName: DanneP
ThreadText: # How does the discount factor (gamma) and memory replay work in Deep Q-learning?
  
  (Asked by: username_0 on 2022-11-18)
  
  Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.
  
  **Correct me if I'm wrong (1):**  
  To train a Deep Q-learning network, every N:th step of action taken, a batch of random samples from replay memory are fed into the model's training function. These samples are totally unrelated to each other, they are unordered and we can't tell if a particular sample is taken from an episode which ended with a positive or negative reward. This is all there is to the training part of a Deep Q-learning network.
  
  From what I've read, the meaning of the discount factor (gamma) is to decide "how much we value future rewards."
  
  **Correct me if I'm wrong (2):**  
  With gamma = 0: The model only cares about the reward from when the state goes from A to B given the action C, and the training is encouraging this behaviour of the model with the reward.   
  With gamma = 0.9: The model cares about future rewards and acts based on what might come 5 (or what number we might want) steps from *this*.
  
  I know I have gotten something wrong somewhere (or on multiple places :D): How can this small value, gamma, make the model train itself "for the future" when nobody (not even human) kan tell what the reward 5 steps from a particular step will be?
  
  To me it would have made sense if we had not only 32 random tuples of (state, action, reward, state') but also a "episode end reward" and a "steps until reward" for each tuple and we in some way put these two new pieces of information into the function.
  
  I'd really like to understand what the gamma does and how. I'm far from a maths person but from the code I've seen in a few tutorials and examples, I can't even begin to understand how this 0.9 or 0.99 can enable the model to train for future reward.
  
  Please help :)
  
  **Tags:** reinforcement-learning, dqn
  
  > **Comments:**
  > **username_1**: You may need to show some of the equations you are looking at. You refer multiple times to a variable called "sigma", but I am not familiar with a use for a variable labelled with $\sigma$ or $\Sigma$ in the usual equations of reinforcement learning or deep Q learning.  
  > **username_0**: Hello again @username_1, I'm sorry, as I wrote I suddenly started to name the factor "sigma" when it should have been "gamma". I've now corrected the original post.   Again, I'm sorry for the mistake, my only excuse being that I was to tired after trying to fix my neural net all day.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-20. Score: 1
  
  Let's try to understand what actually happens when learning. You start with random values for Q(s,a), the function that estimates the reward of an action given a state s, and you then get an immediate reward and then your loss is difference between the given reward + gamma \*future estimated reward, i.e. $loss = Q(s,a) - (reward + gamma \* Q(s', max\_a))$.
  
  Now if gamma=0, it means the loss would not be dependent on what happens afterwards - since the loss would simply be $Q(s,a) = reward$. So eventually Q(s,a) would simply converge to be the immediate reward of action a in state s. This means the agent would not end up learning the optimal strategy, since it would choose the action that maximizes the immediate reward without caring what happens afterwards, even if the step after it will die.
  
  In order to get some intuition on how in practice it actually helps the agent look at the future, try to imagine how this would work out in practice. In the beginning rewards would be random, and at some point it will get to a state before the final episode reward would be very positive if the agent won. Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state - in which case since gamma > 0 it would optimize Q(s,a) to something positive. After millions of s,a pairs you can imagine how this will create a good agent.
  
  Regarding your other suggestion - I am not an expert but I imagine there might be ways to take into account the final episode reward in a different manner - but the main nice thing about DQN is that it is simple and local - and even though you just update based on samples from local actions, with time the network will (hopefully) converge to be optimal.
  
  > **Comments:**
  > **username_0**: Thank you for your reply! I had to let this problem go for awhile so I haven't read your answer until now.  It is this part "Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state" that I have a hard time to intuitively understand. I'll try to read it a few more times and think about it.  It just sounds so weird that it can learn something without knowing the state's position in the chain of events. :)  
  
  ---
  
  