Answers.Body: <p>I see at least 3 issues with your DQN code that need to be fixed:</p>
  
  <ol>
  <li><p>You should not have separate replay memories for successes/failures. Put all of your experiences in one replay memory and sample from it uniformly.</p></li>
  <li><p>Your replay memory is extremely small with only 2,000 samples. You need to make it significantly larger; try at least 100,000 up to 1,000,000 samples.</p></li>
  <li><p>Your <code>batch_target</code> is incorrect. You need to train on <em>returns</em> and not just rewards. In your <code>train</code> function, compute the 1-step return <span class="math-container">$r + \gamma \cdot max_{a'} Q(s',a')$</span>, remembering to set <span class="math-container">$max_{a'} Q(s',a') = 0$</span> if <span class="math-container">$s'$</span> is terminal, and then pass it to <code>model.fit()</code> as your prediction target.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-22T10:27:27.187Z
Answers.Text: Why should you sample uniformly from all memories? In the Cartpole environment I've experienced catastrophic forgetting every time I did it and the solution I did (which worked) was to only sample from the last `x` steps from every failing episode.
Answers.UserDisplayName: asaf92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T11:07:00.373Z
Answers.Text: The biggest remaining issue is probably this: "I just used the state as the input to the NN". Most neural networks are going to have poor performance handling unless the state values are in a nice range.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T19:05:55.900Z
Answers.Text: @NeilSlater I think is an issue. I will normalize the states.
Answers.UserDisplayName: kosa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T19:12:41.450Z
Answers.Text: @Brett I separated them because of two reasons: (1) We get a reward = 1.0 only if we are successful. Until we have enough data on successes, the updates on Q(s, a) are redundant.   (2) Moreover, the distribution of successes vs failures is uneven. However, in the end I just sampled from these bins with replacement, to a new bin, and trained on the Q function using the data in the new bin.  PS: I have tried your way, it was not working at all. So, I came up with this implementation.
Answers.UserDisplayName: kosa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T19:19:12.407Z
Answers.Text: @Brett I think I am following the same update rule. Please see the "test_fun" method in the Q_learn class.
Answers.UserDisplayName: kosa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:34:32.397Z
Answers.Text: @kosa Even though those updates are redundant, they still need to be put into the replay memory because they change the probability of those samples being chosen. It's OK if they're uneven because they reflect the true probabilities of encountering those (s,a,r,s') transitions in the environment. This makes sure your Q-function is an unbiased estimate of the true returns.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:40:59.830Z
Answers.Text: @kosa Your "test_fun" method looks correct now. If you make my other proposed changes and it is still not working, then it's possible you need to make additional changes like what NeilSlater suggested, or there could still be other bugs. Also keep in mind that DQN will not be successful on all environments. Maybe you should test your code on CartPole and see if it can solve that to help you debug.
Answers.UserDisplayName: Brett Daley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T00:28:46.467Z
Answers.LastActivityDate: 2020-02-21T00:28:46.467Z
Answers.UserDisplayName: Brett Daley
Body: <p>I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy)
  <span class="math-container">$$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$</span> </p>
  
  <p>I am using an epsilon greedy policy. 
  In this environment, we get a reward only if we succeed. So I explored with 100% until I have 50 successes. Then I saved the data of failures and success in different bins. Then I sampled (with replacement) from these bins and used them to train the Q network. However, no matter how long I train the agent doesn't seem to learn.</p>
  
  <p>The code is available in <a href="https://colab.research.google.com/drive/1qNsSuUzu6Q4tvQ2d4ZoXkhpQIPaUXjm_" rel="nofollow noreferrer">Colab</a>. I am doing this for a couple of days.</p>
  
  <p>PS: I modified the code for SARSA and Expected SARSA; nothing works.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T21:51:46.380Z
Comments.Text: In general it is quite hard to do code reviews and bug hunts, so you may not get any response. To improve your chances you should give more context on the site, so your question does not rely on people going off site and getting involved in your project. For instance, what diagnostics have you collected, and what are the results? Have you observed more than "the agent doesn't seem to learn" that is worth sharing? What is your state representation for input to the NN, and what its the rough NN architecture - layer sizes, output activation function, loss function etc
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T22:11:50.093Z
Comments.Text: Frozen lake is somewhat harder to solve than expected since it's a very stochastic environment. You will have a hard time solving it with DQN. Try solving it with regular Q learning. Epsilon-greedy isn't ideal because you will get stuck in local optima and you won't get out since exploration will diminish. Try using UCB strategy which is generally better than e-greedy for tabular methods. Also, resetting counters completely after certain amount of episodes in UCB helps in this case.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T22:32:30.027Z
Comments.Text: @Brale_ I tried normal Q learning. It did not train. In any case i will also try UCB.
Comments.UserDisplayName: kosa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T22:40:25.620Z
Comments.Text: @NeilSlater I  used NN with 2-hidden layers (size = 50)+'relu' activations which outputs 4 values(one for each action). I used Adam with a constant learning rate for minimizing MSE error. I did not use any features to represent the state. I just used the state as the input to the NN.
Comments.UserDisplayName: kosa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-21T11:52:36.557Z
Comments.Text: @kosa If your regular Q learning algorithm does not work, then your DQN has no chance to work either. The neural network in DQN is just a storage device to replace the table for the Q-factors if you run out of memory. My hunch is that your `target` in `test_fun` is the real culprit. Currently it's just $Q^{new}$, it should be $(1-\alpha)Q^{old}+\alpha Q^{new}$. Also, frozen lake is a very simple problem (compared to Starcraft or Go), you won't need 2 hidden layers.
Comments.UserDisplayName: Hai Nguyen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-21T19:05:07.373Z
Comments.Text: @HaiNguyen, I will try with this update rule. I make a wide network with a single layer.
Comments.UserDisplayName: kosa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-23T21:43:59.827Z
Comments.Text: @HaiNguyen "If your regular Q-learning algorithm does not work, then your DQN has no chance to work either." Technically this isn't true. For example, consider Atari 2600 games like Pong that DQN can play but Q-learning cannot.
Comments.UserDisplayName: Brett Daley
Tags: <reinforcement-learning><tensorflow><dqn><gym><sarsa>
Title: Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?
UserDisplayName: kosa
ThreadText: # Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?
  
  (Asked by: username_0 on 2020-02-20)
  
  I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy)
  $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max\_{a'}Q(s',a')-Q(s,a))$$
  
  I am using an epsilon greedy policy.
  In this environment, we get a reward only if we succeed. So I explored with 100% until I have 50 successes. Then I saved the data of failures and success in different bins. Then I sampled (with replacement) from these bins and used them to train the Q network. However, no matter how long I train the agent doesn't seem to learn.
  
  The code is available in [Colab](https://colab.research.google.com/drive/1qNsSuUzu6Q4tvQ2d4ZoXkhpQIPaUXjm_). I am doing this for a couple of days.
  
  PS: I modified the code for SARSA and Expected SARSA; nothing works.
  
  **Tags:** reinforcement-learning, tensorflow, dqn, gym, sarsa
  
  > **Comments:**
  > **username_1**: In general it is quite hard to do code reviews and bug hunts, so you may not get any response. To improve your chances you should give more context on the site, so your question does not rely on people going off site and getting involved in your project. For instance, what diagnostics have you collected, and what are the results? Have you observed more than "the agent doesn't seem to learn" that is worth sharing? What is your state representation for input to the NN, and what its the rough NN architecture - layer sizes, output activation function, loss function etc  
  > **username_2**: Frozen lake is somewhat harder to solve than expected since it's a very stochastic environment. You will have a hard time solving it with DQN. Try solving it with regular Q learning. Epsilon-greedy isn't ideal because you will get stuck in local optima and you won't get out since exploration will diminish. Try using UCB strategy which is generally better than e-greedy for tabular methods. Also, resetting counters completely after certain amount of episodes in UCB helps in this case.  
  > **username_0**: @Brale_ I tried normal Q learning. It did not train. In any case i will also try UCB.  
  > **username_0**: @username_1 I  used NN with 2-hidden layers (size = 50)+'relu' activations which outputs 4 values(one for each action). I used Adam with a constant learning rate for minimizing MSE error. I did not use any features to represent the state. I just used the state as the input to the NN.  
  > **username_3**: @username_0 If your regular Q learning algorithm does not work, then your DQN has no chance to work either. The neural network in DQN is just a storage device to replace the table for the Q-factors if you run out of memory. My hunch is that your `target` in `test_fun` is the real culprit. Currently it's just $Q^{new}$, it should be $(1-\alpha)Q^{old}+\alpha Q^{new}$. Also, frozen lake is a very simple problem (compared to Starcraft or Go), you won't need 2 hidden layers.  
  > **username_0**: @username_3, I will try with this update rule. I make a wide network with a single layer.  
  > **username_4**: @username_3 "If your regular Q-learning algorithm does not work, then your DQN has no chance to work either." Technically this isn't true. For example, consider Atari 2600 games like Pong that DQN can play but Q-learning cannot.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2020-02-21. Score: 0
  
  I see at least 3 issues with your DQN code that need to be fixed:
  
  1. You should not have separate replay memories for successes/failures. Put all of your experiences in one replay memory and sample from it uniformly.
  2. Your replay memory is extremely small with only 2,000 samples. You need to make it significantly larger; try at least 100,000 up to 1,000,000 samples.
  3. Your `batch_target` is incorrect. You need to train on *returns* and not just rewards. In your `train` function, compute the 1-step return $r + \gamma \cdot max\_{a'} Q(s',a')$, remembering to set $max\_{a'} Q(s',a') = 0$ if $s'$ is terminal, and then pass it to `model.fit()` as your prediction target.
  
  > **Comments:**
  > **username_1**: The biggest remaining issue is probably this: "I just used the state as the input to the NN". Most neural networks are going to have poor performance handling unless the state values are in a nice range.  
  > **username_0**: @username_1 I think is an issue. I will normalize the states.  
  > **username_0**: @username_4 I separated them because of two reasons: (1) We get a reward = 1.0 only if we are successful. Until we have enough data on successes, the updates on Q(s, a) are redundant.   (2) Moreover, the distribution of successes vs failures is uneven. However, in the end I just sampled from these bins with replacement, to a new bin, and trained on the Q function using the data in the new bin.  PS: I have tried your way, it was not working at all. So, I came up with this implementation.  
  > **username_0**: @username_4 I think I am following the same update rule. Please see the "test_fun" method in the Q_learn class.  
  > **username_4**: @username_0 Even though those updates are redundant, they still need to be put into the replay memory because they change the probability of those samples being chosen. It's OK if they're uneven because they reflect the true probabilities of encountering those (s,a,r,s') transitions in the environment. This makes sure your Q-function is an unbiased estimate of the true returns.  
  > **username_4**: @username_0 Your "test_fun" method looks correct now. If you make my other proposed changes and it is still not working, then it's possible you need to make additional changes like what username_1 suggested, or there could still be other bugs. Also keep in mind that DQN will not be successful on all environments. Maybe you should test your code on CartPole and see if it can solve that to help you debug.  
  > **username_5**: Why should you sample uniformly from all memories? In the Cartpole environment I've experienced catastrophic forgetting every time I did it and the solution I did (which worked) was to only sample from the last `x` steps from every failing episode.  
  
  ---
  
  