Body: <p>I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target.</p>
  
  <p>I have implemented a number of state-of-art techinques to try to improve the ANN performance. Such techniques are: PER, Double DQN, adaptive discount factor, sparse reward. I have also tried Dueling DQN but it performed poorly. I have also tried a number of ANN architectures and it looks like that 2 hidden layers with 128 neurons is the best one so far. My input layer is 12 neurons, the output 10 neurons.</p>
  
  <p>However, as you can see from the image down here, at a certain point the DQN stops learning and gets stuck at around 80% of success rate. I don't understand why it gets stuck, because in my opinion we could reach an higher success rate, 90% at least, but I just can't get out of that "local minimum".</p>
  
  <p>so, my question is: <strong>What are some techniques I can try to unstuck a DQN from something that looks like a local minimum?</strong></p>
  
  <p>figure:</p>
  
  <p><a href="https://i.sstatic.net/LZ5jz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LZ5jz.png" alt="enter image description here"></a></p>
  
  <p><strong><em>note:</strong> the success rate in this picture is computed as the number of successes in the last 100 runs.</em></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T10:32:29.193Z
Comments.Text: Is your success rate measured during training runs, or a separate assessment using greedy actions only? What is your exploration technique - if $\epsilon$-greedy, what is $\epsilon$?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T12:36:08.770Z
Comments.Text: Hi @Neil Slater. It is measured during training. My exploration rule is epsilon greedy with espilon starting at 1 and decaying to 0.01 with a decay factor of 0.9995
Comments.UserDisplayName: olinarr
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T17:41:55.757Z
Comments.Text: Thanks for that detail. It is not possible to say what variation caused by a 0.01 exploration rate would make to your environment. Could you please test one later (mid-80% ) success rate agents, using $\epsilon = 0$, with, say 1000 runs, and report the success rate? If it is close to 80%, then the environment and/or agent are robust to mistakes. If it is notably higher, then this is a better/true measure of the performance of your agent after training
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T17:51:19.300Z
Comments.Text: yes already tested that with 10k runs: 78% success rate with eps = 0. I'm sorry but I didn't understand the rest of the comment: could you explain again?
Comments.UserDisplayName: olinarr
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T17:55:59.163Z
Comments.Text: I meant that the measurement is necessary, because even a small exploration rate can adversely affect success during training. This is very dependent on the nature of the environment. A "robust" (my term, not standard) environment here is one where one or two mistakes don't matter much and can be corrected. Which in turn means that your graph during training is more usable to help rack possible fixes
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><dqn><deep-rl>
Title: DQN Agent not learning anymore - what can I do to fix this?
UserDisplayName: olinarr
ThreadText: # DQN Agent not learning anymore - what can I do to fix this?
  
  (Asked by: username_0 on 2019-04-22)
  
  I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target.
  
  I have implemented a number of state-of-art techinques to try to improve the ANN performance. Such techniques are: PER, Double DQN, adaptive discount factor, sparse reward. I have also tried Dueling DQN but it performed poorly. I have also tried a number of ANN architectures and it looks like that 2 hidden layers with 128 neurons is the best one so far. My input layer is 12 neurons, the output 10 neurons.
  
  However, as you can see from the image down here, at a certain point the DQN stops learning and gets stuck at around 80% of success rate. I don't understand why it gets stuck, because in my opinion we could reach an higher success rate, 90% at least, but I just can't get out of that "local minimum".
  
  so, my question is: **What are some techniques I can try to unstuck a DQN from something that looks like a local minimum?**
  
  figure:
  
  [![enter image description here](https://i.sstatic.net/LZ5jz.png)](https://i.sstatic.net/LZ5jz.png)
  
  ***note:*** the success rate in this picture is computed as the number of successes in the last 100 runs.
  
  **Tags:** deep-learning, dqn, deep-rl
  > **Comments:**
  > **username_1**: Is your success rate measured during training runs, or a separate assessment using greedy actions only? What is your exploration technique - if $\epsilon$-greedy, what is $\epsilon$?  
  > **username_0**: Hi @Neil Slater. It is measured during training. My exploration rule is epsilon greedy with espilon starting at 1 and decaying to 0.01 with a decay factor of 0.9995  
  > **username_1**: Thanks for that detail. It is not possible to say what variation caused by a 0.01 exploration rate would make to your environment. Could you please test one later (mid-80% ) success rate agents, using $\epsilon = 0$, with, say 1000 runs, and report the success rate? If it is close to 80%, then the environment and/or agent are robust to mistakes. If it is notably higher, then this is a better/true measure of the performance of your agent after training  
  > **username_0**: yes already tested that with 10k runs: 78% success rate with eps = 0. I'm sorry but I didn't understand the rest of the comment: could you explain again?  
  > **username_1**: I meant that the measurement is necessary, because even a small exploration rate can adversely affect success during training. This is very dependent on the nature of the environment. A "robust" (my term, not standard) environment here is one where one or two mistakes don't matter much and can be corrected. Which in turn means that your graph during training is more usable to help rack possible fixes  
  
  ---
  
  ## 0 answers
  
  