Answers.Body: <p>I was attaining negative values out of my convolutional layers and then using relu on them resulted in the gradient of the activation being 0. Hence, my Q values were not being updated. I've since updated my activations to be ELU. Thanks for the help.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-18T06:47:38.773Z
Answers.LastActivityDate: 2018-09-18T06:47:38.773Z
Answers.UserDisplayName: rtz
Body: <p>I am trying to use Deep-Q learning environment to learn Super Mario Bros. <a href="https://github.com/rtang23/FYP/blob/5abfccf3e86742b8487080493789e13b6f812388/training2.py" rel="nofollow noreferrer">The implementation is on Github</a>.</p>
  
  <p>I have a neural network that Q values update within an episode for a very small learning rate (0.00005). However, even if I increase the learning rate to 0.00025, the Q values do not change within an episode as they are predicting the same Q values regardless of what state it is in. For example, if Mario moves right, the Q value is the same. When I start a new episode, the Q values change though.</p>
  
  <p>I think that the Q values should be changing within an episode as the game should be seeing different parts and taking different actions. Why don't I observe this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T07:41:52.833Z
Comments.Text: Learning rate should only have an effect on the **magnitude** of updates to your Q-values, it should not influence **when** updates are triggered. What makes you think that it does? If you are certain that you have correctly observed this phenomenon, where did you get the code from that does this?
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T07:50:48.350Z
Comments.Text: @DennisSoemers if I had a larger learning rate, would the magnitude of updates to my Q-Values be smaller or bigger? As in my own code, the higher learning rate results in the Q-values not updating or staying the same. I've done something wrong, was just trying to find the root of it
Comments.UserDisplayName: rtz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T07:51:17.050Z
Comments.Text: @NeilSlater the environment is Super Mario Bros. I am using Deep-Q learning
Comments.UserDisplayName: rtz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T07:54:25.480Z
Comments.Text: @rtz Typically, we compute some sort of error (something like `target_output - current_output`), multiply that with the learning rate, and then update accordingly (that's the basic idea). So, a larger learning rate should result in larger updates. A learning rate of $0$ would mean no updating whatsoever. All of this is of course you mean the same thing as I do when you say "learning rate"... though I don't think it's really an ambiguous term generally
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T08:01:10.610Z
Comments.Text: @DennisSoemers definitely talking about the same thing and that's what I'm doing but unsure what could be going wrong. If you were really interested, I could send you the code but that definitely helps. Thank you
Comments.UserDisplayName: rtz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T08:05:16.393Z
Comments.Text: @rtz You can put the code somewhere and I'll probably have time somewhere today to have a quick look through it (though not right now). Inspecting a DQN implementation line-by-line would probably be a lot of work, but can at least see if I happen to notice something in a quick glance.
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T08:06:47.577Z
Comments.Text: @DennisSoemers if you don't have the time it's okay too. Thank you very much!
  https://github.com/rtang23/FYP/blob/5abfccf3e86742b8487080493789e13b6f812388/training2.py
Comments.UserDisplayName: rtz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T12:39:39.500Z
Comments.Text: Can you edit the question to elaborate in more detail exactly what issue you are observing? What do you mean when you say that "Q values do not change within an episode"? Do you mean that the network's parameters are not being updated? Or do you mean that it keeps predicting the same values regardless of what state it's in (or is it maybe stuck in the same state)? Exactly what is it that you observe?
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T13:04:21.300Z
Comments.Text: @DennisSoemers I am thinking the learning rate would not affect what state it is in. However, that does seem to be the case. I also checked with my code if it stayed in the same state when this happened and this was not the case.
Comments.UserDisplayName: rtz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T08:09:05.547Z
Comments.Text: I didn't really see anything strange in your code. I can say certainly that modifying the learning rate should not have the kinds of effects you're describing. That said, it's still not 100% clear **exactly** what you are observing though, my bet is on a misunderstanding/misinterpretation there. Before I can try to find an explanation for what you observe, I really need to know every single tiny detail of exactly what it is you are observing.
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-17T13:39:47.113Z
Comments.Text: @DennisSoemers sorry for the delayed reply. So what is happening is when I use a learning rate of 0.00025, the Q values throughout the episode stay the same. I.e. if Mario is at the start of the screen and moves to the right, the Q values are not being updated as I am printing them constantly in a test episode. I hope that clears it up
Comments.UserDisplayName: rtz
Tags: <neural-networks><machine-learning><deep-learning><reinforcement-learning><game-ai>
Title: Should Q values be changing within an epoch/episode or should they change after one episode/epoch?
UserDisplayName: rtz
ThreadText: # Should Q values be changing within an epoch/episode or should they change after one episode/epoch?
  
  (Asked by: username_0 on 2018-09-12)
  
  I am trying to use Deep-Q learning environment to learn Super Mario Bros. [The implementation is on Github](https://github.com/rtang23/FYP/blob/5abfccf3e86742b8487080493789e13b6f812388/training2.py).
  
  I have a neural network that Q values update within an episode for a very small learning rate (0.00005). However, even if I increase the learning rate to 0.00025, the Q values do not change within an episode as they are predicting the same Q values regardless of what state it is in. For example, if Mario moves right, the Q value is the same. When I start a new episode, the Q values change though.
  
  I think that the Q values should be changing within an episode as the game should be seeing different parts and taking different actions. Why don't I observe this?
  
  **Tags:** neural-networks, machine-learning, deep-learning, reinforcement-learning, game-ai
  
  > **Comments:**
  > **username_1**: Learning rate should only have an effect on the **magnitude** of updates to your Q-values, it should not influence **when** updates are triggered. What makes you think that it does? If you are certain that you have correctly observed this phenomenon, where did you get the code from that does this?  
  > **username_0**: @username_1 if I had a larger learning rate, would the magnitude of updates to my Q-Values be smaller or bigger? As in my own code, the higher learning rate results in the Q-values not updating or staying the same. I've done something wrong, was just trying to find the root of it  
  > **username_0**: @NeilSlater the environment is Super Mario Bros. I am using Deep-Q learning  
  > **username_1**: @username_0 Typically, we compute some sort of error (something like `target_output - current_output`), multiply that with the learning rate, and then update accordingly (that's the basic idea). So, a larger learning rate should result in larger updates. A learning rate of $0$ would mean no updating whatsoever. All of this is of course you mean the same thing as I do when you say "learning rate"... though I don't think it's really an ambiguous term generally  
  > **username_0**: @username_1 definitely talking about the same thing and that's what I'm doing but unsure what could be going wrong. If you were really interested, I could send you the code but that definitely helps. Thank you  
  > **username_1**: @username_0 You can put the code somewhere and I'll probably have time somewhere today to have a quick look through it (though not right now). Inspecting a DQN implementation line-by-line would probably be a lot of work, but can at least see if I happen to notice something in a quick glance.  
  > **username_0**: @username_1 if you don't have the time it's okay too. Thank you very much! https://github.com/rtang23/FYP/blob/5abfccf3e86742b8487080493789e13b6f812388/training2.py  
  > **username_1**: Can you edit the question to elaborate in more detail exactly what issue you are observing? What do you mean when you say that "Q values do not change within an episode"? Do you mean that the network's parameters are not being updated? Or do you mean that it keeps predicting the same values regardless of what state it's in (or is it maybe stuck in the same state)? Exactly what is it that you observe?  
  > **username_0**: @username_1 I am thinking the learning rate would not affect what state it is in. However, that does seem to be the case. I also checked with my code if it stayed in the same state when this happened and this was not the case.  
  > **username_1**: I didn't really see anything strange in your code. I can say certainly that modifying the learning rate should not have the kinds of effects you're describing. That said, it's still not 100% clear **exactly** what you are observing though, my bet is on a misunderstanding/misinterpretation there. Before I can try to find an explanation for what you observe, I really need to know every single tiny detail of exactly what it is you are observing.  
  > **username_0**: @username_1 sorry for the delayed reply. So what is happening is when I use a learning rate of 0.00025, the Q values throughout the episode stay the same. I.e. if Mario is at the start of the screen and moves to the right, the Q values are not being updated as I am printing them constantly in a test episode. I hope that clears it up  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-09-18. Score: 2
  
  I was attaining negative values out of my convolutional layers and then using relu on them resulted in the gradient of the activation being 0. Hence, my Q values were not being updated. I've since updated my activations to be ELU. Thanks for the help.
  
  
  ---
  
  