Answers.Body: <p>You report that your model is configured as follows:</p>
  <pre><code>model = keras.models.Sequential()
  model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
  model.add(tf.keras.layers.Dense(64))
  model.add(tf.keras.layers.Dense(32))
  model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
  
  opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
  model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
  return model
  </code></pre>
  <p>It is important to note that the TensorFlow/Keras <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense" rel="nofollow noreferrer">documentation</a> reports regarding the <code>activation</code> keyword parameter:</p>
  <blockquote>
  <p>[The activation parameter allows you to specify the] Activation function to use. If you don't specify anything, no activation is applied (ie. &quot;linear&quot; activation: a(x) = x).</p>
  </blockquote>
  <p>Currently, your network is using the default of <code>None</code>, which is equivalent to <code>linear</code>.  Having multiple layers but only using a <code>linear</code> activation is akin to performing multiple linear transformations in series; the entire series is <em>always</em> collapsible to a single transformation matrix.</p>
  <p>In other words, since all of your layers are linear, the network does not really gain complexity by adding additional layers.  In effect you have only one hidden layer.</p>
  <p>You might try simply adding a non-linear activation.  Perhaps:</p>
  <pre><code>model = keras.models.Sequential()
  model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
  model.add(tf.keras.layers.Dense(64, activation='relu'))
  model.add(tf.keras.layers.Dense(32, activation='relu'))
  model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
  
  opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
  model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
  return model
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T23:21:12.077Z
Answers.LastActivityDate: 2022-03-24T23:21:12.077Z
Answers.UserDisplayName: David Hoelzer
Body: <p>I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior get better and better as time goes on? I tried training 3 times and the same thing happened each time. Letting the model run longer never saw the model improve again and basically stayed random. I'm new to data science/machine learning and if someone could point me towards some resources to prevent this behavior I would very much appriciate it.</p>
  <p>For some more context, I'm trying to solve the cart pole problem (<a href="https://en.wikipedia.org/wiki/Inverted_pendulum#:%7E:text=An%20inverted%20pendulum%20is%20a,additional%20help%20will%20fall%20over." rel="nofollow noreferrer">inverted pendulum</a>) using Q-Learning. The graph I've included below shows how long the model is able to balance the pendulum. The x-axis represents the episodes and the y-axis represents time balanced.</p>
  <p><a href="https://i.sstatic.net/Z2bF0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Z2bF0.png" alt="enter image description here" /></a></p>
  <p><strong>EDIT 1:</strong></p>
  <p>When the graph was generated, this is how the model was created. I never set a learning rate.</p>
  <pre><code>model = keras.models.Sequential()
  model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
  model.add(tf.keras.layers.Dense(64))
  model.add(tf.keras.layers.Dense(32))
  model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
      
  model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
  return model
  </code></pre>
  <p>I've since changed it to this.</p>
  <pre><code> model = keras.models.Sequential()
   model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
   model.add(tf.keras.layers.Dense(64))
   model.add(tf.keras.layers.Dense(32))
   model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
  
  opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
  model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
  return model
  </code></pre>
  <p>I have a 'target model' that gets updated every 5 'episodes.' An episode ends after 15s or when the angle is greater than 45 degrees.
  Since I've changed it to use the <code>tf.keras.optimizers.Adam</code> optimizer, the model progress looks very noisy. Since changing the optimizer, when I evaluate it while training, sometimes the agent will be very good then drop the to very bad the next time it is evaluated. (The agent is evaluated after training on 20 episodes. For an evaluation, the agent runs through 10 episodes and the time is averaged). Here is a plot of the training progress. I've tried changing the learning rate to be lower but the model doesn't appear to improve at all.</p>
  <p><a href="https://i.sstatic.net/Hzgki.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Hzgki.png" alt="enter image description here" /></a></p>
  <p>Would an evolutionary type algorithm be better because it takes the best of each generation? I feel like that wouldn't allow a model to drop like it does now. However, I would still like to know why what I'm doing now isn't working.</p>
  <p><strong>EDIT 2:</strong></p>
  <p>I've since changed things to use a learning rate of 0.00003 (3e-5) as suggested in the comments. I've also changed the target model to update every 1000 episodes. I'm still finidng similar results. The training progress graph is below (like before it has episodes on the x-axis and average time survived on the y-axis). I am training the model on every step of the episode, should I only be training after an episode completes?</p>
  <p><a href="https://i.sstatic.net/xsjgy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xsjgy.png" alt="Training EDIT3" /></a>
  The green vertical lines represent when the target model was updated. The red horizontal lines represent the min and max times. Ignore the x-axis. Model progress was recorded every 20 episodes so the x-axis should be scaled by 20.</p>
  <p><strong>EDIT 3:</strong></p>
  <p>I'm solving the cart pole problem. I want to reward behavior when the pole's angle is closer to upright (theta=0). I also want to reward behavior when the cart itself is closer to the center (x=0). I take away 150 from the reward when it tips over.</p>
  <pre><code>def get_reward(self):
      score = 0.0
      fail = False
      complete = False
  
      if abs(self.cart.theta) &gt; CartEnv.THETA_MAX: # THETA_MAX = 45 degrees or pi/4 radians.
          fail = True
          complete = True
          score -= 150
  
      if not fail and self.time &gt;= 15:
          complete = True
  
      # Take away score further from center
      score += 30 - (abs(self.cart.theta) * RAD_TO_DEG) # [30, -15]
      score += .5 * max(10 - abs(self.cart.x), -20) # [5, -10]
  
      return (score, complete)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-17T09:46:15.680Z
Comments.Text: Please, edit your post to provide more details about the hyper-parameters that you're using (e.g. learning rate, number of layers, etc.). Those might be useful to answer your question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-18T01:58:49.233Z
Comments.Text: @nbro , I've added an edit. I'm new to the world of data science/machine learning so any advice would be appriciated!
Comments.UserDisplayName: Josh Majors
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-18T09:04:35.820Z
Comments.Text: I would choose a _learning rate_ way smaller. Most of implementations that I have seen assume something around 3e-5. Also, the rate that you're updating the target net seems too short. DQN are know to be unstable and the idea of use a target net aims circumvent it. Perhaps try to use something above ~1000 and let me know if your agent improved.
Comments.UserDisplayName: HenDoNR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-23T12:11:10.087Z
Comments.Text: @HenDoNR, under 'EDIT 2' you can see the results from the changes you suggested. The model appears to make changes after the target model is updated. Should I update the target model only when the model appears to be doing well?
Comments.UserDisplayName: Josh Majors
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T13:17:31.103Z
Comments.Text: Thanks to give more details. You definitely shouldn't train your agent after the end of each episode. The idea of Experience Replay/Replay memory is store the experiences and shuffled them to break the correlation between data. It's always hard to put parameters in numbers, but try to increase substantially the amount of experiences stored before train your agent. I would say the same for you target net update rate
Comments.UserDisplayName: HenDoNR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T17:24:49.957Z
Comments.Text: @HenDoNR, thank you for your help! But ok good, I have been training my model on each step of the episode, not just the end. I wait for my replay memory to have >1,000 transitions before doing any training whatsover. Should I wait until I have >10,000? Is that a good number? Everytime I train I select a random sample of 64 samples from my replay memory. Should I be using more than 64 samples? Could another issue be my reward function? I've posted my reward function under EDIT 3.
Comments.UserDisplayName: Josh Majors
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T19:00:37.223Z
Comments.Text: Is that the actual model you are using? If so, you have another issue that will significantly hinder its ability to learn.  If you add a `layers.Dense(32)` without specifying an activation, the default activation is `linear`.  Given that, creating multiple layers will have no benefit for your model since it will trivially collapse to a single linear transformation.  You likely want to specify *something* as your activation.  Perhaps start with `relu`?
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T19:04:58.570Z
Comments.Text: I just checked an [implementation](https://towardsdatascience.com/deep-q-learning-for-the-cartpole-44d761085c2f) and your parameters seems okay. Indeed they used a smaller update target rate and higher learning rate than I suggested you. Other things that might help you is ReLu in dense layers and set the discount rate gamma =~0.99.  Furthermore, check if your epsilon decay is progressing towards fully rely on Q-function (and this is quite important!)
Comments.UserDisplayName: HenDoNR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T23:04:59.440Z
Comments.Text: @JoshMajors if you are just asking "why", the answer is that in RL "learning collapses" are very common. Even when using algorithms like TRPO and PPO (*which are specifically designed to prevent learning collapses*), they are still rather common! Welcome to RL
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T23:06:47.857Z
Comments.Text: @DavidHoelzer , that makes a lot of sense with the activation function. That was a huge question I had with machine learning in general: how can we get nonlinear behavior when the system is just a bunch of linear equations? And the answer seems to be the activation functions, right? Right now I'm training with 'relu' as my activation for the hidden layers. I made my activation 'tanh' for the final layer. Will update with results.
Comments.UserDisplayName: Josh Majors
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T23:12:03.840Z
Comments.Text: @profPlum , I've seen some things on "catastrophic forgetting" but it felt like I was doing all of the things to combat it. I have my replay memory with things from the start of my training, I tried a really low learning rate of 3e-5, and wasn't really sure what was going wrong. In practice do people just train a model until it reaches a peak, save the peak, and retry with changing hyperparameters until you keep getting higher peaks? Or is there someway to completely prevent it so it never drops out?
Comments.UserDisplayName: Josh Majors
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T23:14:56.633Z
Comments.Text: @HenDoNR , right now I'm trying things with a learning rate of 3e-5 and and the target model is being updated every 500 iterations. It seems like adding the activation function is helping. Will report back with the results. Thanks so much for all your help!
Comments.UserDisplayName: Josh Majors
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-24T23:21:37.047Z
Comments.Text: I've converted my comment to an answer.
Comments.UserDisplayName: David Hoelzer
Tags: <reinforcement-learning><machine-learning><q-learning><dqn>
Title: Deep Q-Learning Model Effectiveness Improves then Crashes
UserDisplayName: Josh Majors
ThreadText: # Deep Q-Learning Model Effectiveness Improves then Crashes
  
  (Asked by: username_0 on 2022-03-14)
  
  I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior get better and better as time goes on? I tried training 3 times and the same thing happened each time. Letting the model run longer never saw the model improve again and basically stayed random. I'm new to data science/machine learning and if someone could point me towards some resources to prevent this behavior I would very much appriciate it.
  
  For some more context, I'm trying to solve the cart pole problem ([inverted pendulum](https://en.wikipedia.org/wiki/Inverted_pendulum#:%7E:text=An%20inverted%20pendulum%20is%20a,additional%20help%20will%20fall%20over.)) using Q-Learning. The graph I've included below shows how long the model is able to balance the pendulum. The x-axis represents the episodes and the y-axis represents time balanced.
  
  [![enter image description here](https://i.sstatic.net/Z2bF0.png)](https://i.sstatic.net/Z2bF0.png)
  
  **EDIT 1:**
  
  When the graph was generated, this is how the model was created. I never set a learning rate.
  
  ```
  model = keras.models.Sequential()
  model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
  model.add(tf.keras.layers.Dense(64))
  model.add(tf.keras.layers.Dense(32))
  model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
      
  model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
  return model
  
  ```
  
  I've since changed it to this.
  
  ```
   model = keras.models.Sequential()
   model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
   model.add(tf.keras.layers.Dense(64))
   model.add(tf.keras.layers.Dense(32))
   model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
  
  opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
  model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
  return model
  
  ```
  
  I have a 'target model' that gets updated every 5 'episodes.' An episode ends after 15s or when the angle is greater than 45 degrees.
  Since I've changed it to use the `tf.keras.optimizers.Adam` optimizer, the model progress looks very noisy. Since changing the optimizer, when I evaluate it while training, sometimes the agent will be very good then drop the to very bad the next time it is evaluated. (The agent is evaluated after training on 20 episodes. For an evaluation, the agent runs through 10 episodes and the time is averaged). Here is a plot of the training progress. I've tried changing the learning rate to be lower but the model doesn't appear to improve at all.
  
  [![enter image description here](https://i.sstatic.net/Hzgki.png)](https://i.sstatic.net/Hzgki.png)
  
  Would an evolutionary type algorithm be better because it takes the best of each generation? I feel like that wouldn't allow a model to drop like it does now. However, I would still like to know why what I'm doing now isn't working.
  
  **EDIT 2:**
  
  I've since changed things to use a learning rate of 0.00003 (3e-5) as suggested in the comments. I've also changed the target model to update every 1000 episodes. I'm still finidng similar results. The training progress graph is below (like before it has episodes on the x-axis and average time survived on the y-axis). I am training the model on every step of the episode, should I only be training after an episode completes?
  
  [![Training EDIT3](https://i.sstatic.net/xsjgy.png)](https://i.sstatic.net/xsjgy.png)
  The green vertical lines represent when the target model was updated. The red horizontal lines represent the min and max times. Ignore the x-axis. Model progress was recorded every 20 episodes so the x-axis should be scaled by 20.
  
  **EDIT 3:**
  
  I'm solving the cart pole problem. I want to reward behavior when the pole's angle is closer to upright (theta=0). I also want to reward behavior when the cart itself is closer to the center (x=0). I take away 150 from the reward when it tips over.
  
  ```
  def get_reward(self):
      score = 0.0
      fail = False
      complete = False
  
      if abs(self.cart.theta) > CartEnv.THETA_MAX: # THETA_MAX = 45 degrees or pi/4 radians.
          fail = True
          complete = True
          score -= 150
  
      if not fail and self.time >= 15:
          complete = True
  
      # Take away score further from center
      score += 30 - (abs(self.cart.theta) * RAD_TO_DEG) # [30, -15]
      score += .5 * max(10 - abs(self.cart.x), -20) # [5, -10]
  
      return (score, complete)
  
  ```
  
  **Tags:** reinforcement-learning, machine-learning, q-learning, dqn
  
  > **Comments:**
  > **username_1**: Please, edit your post to provide more details about the hyper-parameters that you're using (e.g. learning rate, number of layers, etc.). Those might be useful to answer your question.  
  > **username_0**: @username_1 , I've added an edit. I'm new to the world of data science/machine learning so any advice would be appriciated!  
  > **username_2**: I would choose a _learning rate_ way smaller. Most of implementations that I have seen assume something around 3e-5. Also, the rate that you're updating the target net seems too short. DQN are know to be unstable and the idea of use a target net aims circumvent it. Perhaps try to use something above ~1000 and let me know if your agent improved.  
  > **username_0**: @username_2, under 'EDIT 2' you can see the results from the changes you suggested. The model appears to make changes after the target model is updated. Should I update the target model only when the model appears to be doing well?  
  > **username_2**: Thanks to give more details. You definitely shouldn't train your agent after the end of each episode. The idea of Experience Replay/Replay memory is store the experiences and shuffled them to break the correlation between data. It's always hard to put parameters in numbers, but try to increase substantially the amount of experiences stored before train your agent. I would say the same for you target net update rate  
  > **username_0**: @username_2, thank you for your help! But ok good, I have been training my model on each step of the episode, not just the end. I wait for my replay memory to have >1,000 transitions before doing any training whatsover. Should I wait until I have >10,000? Is that a good number? Everytime I train I select a random sample of 64 samples from my replay memory. Should I be using more than 64 samples? Could another issue be my reward function? I've posted my reward function under EDIT 3.  
  > **username_3**: Is that the actual model you are using? If so, you have another issue that will significantly hinder its ability to learn.  If you add a `layers.Dense(32)` without specifying an activation, the default activation is `linear`.  Given that, creating multiple layers will have no benefit for your model since it will trivially collapse to a single linear transformation.  You likely want to specify *something* as your activation.  Perhaps start with `relu`?  
  > **username_2**: I just checked an [implementation](https://towardsdatascience.com/deep-q-learning-for-the-cartpole-44d761085c2f) and your parameters seems okay. Indeed they used a smaller update target rate and higher learning rate than I suggested you. Other things that might help you is ReLu in dense layers and set the discount rate gamma =~0.99.  Furthermore, check if your epsilon decay is progressing towards fully rely on Q-function (and this is quite important!)  
  > **username_4**: @username_0 if you are just asking "why", the answer is that in RL "learning collapses" are very common. Even when using algorithms like TRPO and PPO (*which are specifically designed to prevent learning collapses*), they are still rather common! Welcome to RL  
  > **username_0**: @username_3 , that makes a lot of sense with the activation function. That was a huge question I had with machine learning in general: how can we get nonlinear behavior when the system is just a bunch of linear equations? And the answer seems to be the activation functions, right? Right now I'm training with 'relu' as my activation for the hidden layers. I made my activation 'tanh' for the final layer. Will update with results.  
  > **username_0**: @username_4 , I've seen some things on "catastrophic forgetting" but it felt like I was doing all of the things to combat it. I have my replay memory with things from the start of my training, I tried a really low learning rate of 3e-5, and wasn't really sure what was going wrong. In practice do people just train a model until it reaches a peak, save the peak, and retry with changing hyperparameters until you keep getting higher peaks? Or is there someway to completely prevent it so it never drops out?  
  > **username_0**: @username_2 , right now I'm trying things with a learning rate of 3e-5 and and the target model is being updated every 500 iterations. It seems like adding the activation function is helping. Will report back with the results. Thanks so much for all your help!  
  > **username_3**: I've converted my comment to an answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-03-24. Score: 1 | Accepted answer
  
  You report that your model is configured as follows:
  
  ```
  model = keras.models.Sequential()
  model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
  model.add(tf.keras.layers.Dense(64))
  model.add(tf.keras.layers.Dense(32))
  model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
  
  opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
  model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
  return model
  
  ```
  
  It is important to note that the TensorFlow/Keras [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) reports regarding the `activation` keyword parameter:
  
  > [The activation parameter allows you to specify the] Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
  
  Currently, your network is using the default of `None`, which is equivalent to `linear`. Having multiple layers but only using a `linear` activation is akin to performing multiple linear transformations in series; the entire series is *always* collapsible to a single transformation matrix.
  
  In other words, since all of your layers are linear, the network does not really gain complexity by adding additional layers. In effect you have only one hidden layer.
  
  You might try simply adding a non-linear activation. Perhaps:
  
  ```
  model = keras.models.Sequential()
  model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
  model.add(tf.keras.layers.Dense(64, activation='relu'))
  model.add(tf.keras.layers.Dense(32, activation='relu'))
  model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
  
  opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
  model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
  return model
  
  ```
  
  
  ---
  
  