Answers.Body: <p>1) Vary the number of enemies during the training process.
  2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T11:45:43.120Z
Answers.Text: Thank you for really fast responses. I have been stuck on this issue for the last 3 days and now, out of nowhere, it started working. Combination of two things has been critical to making it work: slightly more neurons in the hidden layer + significantly higher learning rate. Changing the learning rate from 0.0001 to 0.01 made it work.
Answers.UserDisplayName: Savco
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T18:27:02.627Z
Answers.Text: @Savco Thanks for the update, it is always useful to know which solutions people have found.
Answers.UserDisplayName: DrMcCleod
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T10:30:27.690Z
Answers.LastActivityDate: 2019-02-10T10:30:27.690Z
Answers.UserDisplayName: DrMcCleod
Body: <p>I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions:</p>
  
  <ol>
  <li><code>do_nothing</code></li>
  <li><code>prepare_for(e)</code></li>
  <li><code>attack(e)</code> </li>
  </ol>
  
  <p>where <code>e</code> is some enemy. </p>
  
  <p>In the case of two enemies, the action vector has 5 elements:</p>
  
  <pre><code>|   0       |      1          |      2      |        3         |     4      |
  -----------------------------------------------------------------------------
  |do_nothing | prepare_for(e1) |  attack(e1) |  prepare_for(e2) | attack(e2) |
  -----------------------------------------------------------------------------
  </code></pre>
  
  <p>After a couple of episodes, the agent always starts picking the first <code>do_nothing</code> action, which is not desired. Changing reward for <code>do_nothing</code> action is not helping, even using significantly higher negative reward, than for other actions. </p>
  
  <p>There is no problem with the environment with only one enemy. (Only using columns 0, 1, 2). I feel like my action encoding can be the issue, but I can't figure it out, how to fix it. Any suggestions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-10T10:59:17.910Z
Comments.Text: You have to provide a bit more details. What is the shape of your neural network? What are you using as input, state features or images? Are you providing state observation + action as the input or only state observation? How does the output look like, are there 5 nodes each for specific action or is it only a single output node for the specific action you provided as a part of the input. What are the rewards for your environment? Are you using exploratory policy? Are you using experience replay? Are you using separate networks? Are you providing enough training time?
Comments.UserDisplayName: Brale
Tags: <deep-learning><reinforcement-learning><deep-rl>
Title: Deep Q-learning is not performing well when there are several enemies
UserDisplayName: Savco
ThreadText: # Deep Q-learning is not performing well when there are several enemies
  
  (Asked by: username_0 on 2019-02-10)
  
  I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions:
  
  do_nothing
  prepare_for(e)
  attack(e) 
  
  where e is some enemy. 
  In the case of two enemies, the action vector has 5 elements:
  |   0       |      1          |      2      |        3         |     4      |
  -----------------------------------------------------------------------------
  |do_nothing | prepare_for(e1) |  attack(e1) |  prepare_for(e2) | attack(e2) |
  -----------------------------------------------------------------------------
  
  After a couple of episodes, the agent always starts picking the first do_nothing action, which is not desired. Changing reward for do_nothing action is not helping, even using significantly higher negative reward, than for other actions. 
  There is no problem with the environment with only one enemy. (Only using columns 0, 1, 2). I feel like my action encoding can be the issue, but I can't figure it out, how to fix it. Any suggestions?
  
  **Tags:** deep-learning, reinforcement-learning, deep-rl
  > **Comments:**
  > **username_1**: You have to provide a bit more details. What is the shape of your neural network? What are you using as input, state features or images? Are you providing state observation + action as the input or only state observation? How does the output look like, are there 5 nodes each for specific action or is it only a single output node for the specific action you provided as a part of the input. What are the rewards for your environment? Are you using exploratory policy? Are you using experience replay? Are you using separate networks? Are you providing enough training time?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-02-10. Score: 1
  
  1) Vary the number of enemies during the training process.
  2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation.
  
  > **Comments:**
  > **username_0**: Thank you for really fast responses. I have been stuck on this issue for the last 3 days and now, out of nowhere, it started working. Combination of two things has been critical to making it work: slightly more neurons in the hidden layer + significantly higher learning rate. Changing the learning rate from 0.0001 to 0.01 made it work.  
  > **username_2**: @Savco Thanks for the update, it is always useful to know which solutions people have found.  
  
  ---
  
  