Answers.Body: <p>Your implementation of single-step Q-learning with neural network and experience replay is basically correct.</p>
  
  <p>There are a few blocking issues preventing you seeing it working correctly.</p>
  
  <p>Your main problem is a bug in your feature scaling routine. That is a Python issue, not really an AI one. In short, you scale the input features in-place multiple times, including an effective double-scaling of <code>next_state</code> (when it gets copied to <code>state</code> you scale it in place a second time in the next loop) so that all the states that you store in the experience replay table never match to any input states. You need to change your definition of <code>scale</code> to not do this. A very simple re-write of your routine would be:</p>
  
  <pre><code>def scale(s):
      return [s[0]/500, s[1]/500, s[2]/300, s[3]/3]
  </code></pre>
  
  <p>In addition, you need to change random action selection to:</p>
  
  <pre><code>np.random.randint(0, 3)
  </code></pre>
  
  <p>because the end of range is never output (this matches behaviour of other range values and operators in Python). Not including the "do nothing" action during exploration means that the agent will test it less, and have less data to work with to assess whether it is the best action. This is a minor issue for this environment, but you should fix it nonetheless.</p>
  
  <blockquote>
    <p>What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). </p>
  </blockquote>
  
  <p>I cannot replicate this fault and can train 20 episodes in around 2.5 minutes - that's over 20 times faster than you report. I am not using a GPU. Possibly in your case, Theano and Pygame are fighting for control of the GPU, or you may have a GPU configuration issue with Theano. Try turning GPU acceleraton off to verify whether it helps. You don't benefit much from a GPU for this environment (most time is spent in Python running the Q-learning and the environment), so can afford to put solving that issue to one side for now.</p>
  
  <blockquote>
    <p>Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern</p>
  </blockquote>
  
  <p>Sadly, I cannot see the output at all on my MacBook pro, but I was able to use feedback of the expected score. A random agent gets a mean score of ~5.5 per episode. With the scale function corrected, and a rough guess at working hyperparameters, I can get an average score of ~17 per episode consistently after 60 episodes of training. After 150 episodes - taking 20 minutes to train - the agent was scoring ~20 per episode and I stopped there. It is possible that an expected score around 20 is already optimal, as it is a very simple environment, but I don't know.</p>
  
  <p>Once you have a working system, there are lots of hyperparameters you could play with to try and improve this. I got my results by making the following changes after fixing the <code>scale</code> function:</p>
  
  <ul>
  <li><p>Starting epsilon of 1.0</p></li>
  <li><p>Repay memory size 10,000</p></li>
  <li><p>Only start learning when replay memory has greater than 1,000 entries</p></li>
  <li><p>Discount factor <span class="math-container">$\gamma$</span> 0.99</p></li>
  <li><p>Neural network with 20 neurons per layer with <code>tanh</code> activation instead of <code>relu</code></p></li>
  </ul>
  
  <p>There is quite a lot else you could change that might make the agent learn more effectively or perhaps aim for a more optimal policy. Have fun experimenting!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-14T21:33:12.270Z
Answers.Text: Thanks for taking the time to try my code! I implemented all that you said. I tried using `tensorflow` instead of `tensorflow-gpu` in order to just use the CPU. It is still just as slow. This loop: `for j in range(len(experiences)):` takes a lot of time I guess. Could this be an IDE issue? I am using Spyder. I also completely removed PyGame from the code (just while training).
Answers.UserDisplayName: shurup
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T08:15:42.897Z
Answers.Text: @NickSolonko: Sorry I cannot tell why you are experiencing such slow times. All I can say is that is not due to your code in general which runs fine on my MacBook (which is 2012 model, so not top of the range for a modern machine). You could vectorise that loop in Keras - if you are not sure how to, ask here or on SO, or on DataScience. Not sure if that will help you that much though, as the core problem seems to be somewhere else.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-17T13:25:17.703Z
Answers.Text: Did it take your system 2.5 minutes even when sampling 1,000 Replay Memory records? I tried another Python IDE on another computer. It is a little faster, but will ask another question on Stack to try to fix the problem.
Answers.UserDisplayName: shurup
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-17T15:13:16.903Z
Answers.Text: @NickSolonko: I didn't try with just 1000, and cannot at the moment. I upgraded it to 10,000 for my test. I doubt my change would make it faster, as the 2.5 minutes was for 20 episodes (and 20 minutes total to reach top performance for the agent), and sampling from 10,000 entries would be slightly slower in Python, probably not noticeable though
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-10T11:53:25.677Z
Answers.LastActivityDate: 2019-08-10T16:44:28.450Z
Answers.LastEditDate: 2019-08-10T16:44:28.450Z
Answers.UserDisplayName: Neil Slater
Body: <p>This is a follow-up question about one I asked earlier. The first question is <a href="https://ai.stackexchange.com/questions/13793/deep-q-learning-algorithm-for-simple-python-game-makes-player-stuck">here</a>. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (gain points) and some is bad (lose points). NN Architecture:</p>
  
  <pre><code>    #inputs - paddle.x, food.x, food.y, food.type
      #moves: left, right, stay
      model = Sequential()
      model.add(Dense(10, input_shape=(4,), activation='relu'))
      model.add(Dense(10, activation='relu'))
      model.add(Dense(3, activation='linear'))
      model.compile(loss='mean_squared_error', optimizer='adam')
  </code></pre>
  
  <p>As suggested in the other question, I scaled my inputs to be between 0 and 1. Also, implemented experience replay (although I am not confident I did it correctly). </p>
  
  <p>Here is my ReplayMemory class:</p>
  
  <pre><code>class ReplayMemory():
      def __init__(self, capacity):
          self.capacity = capacity
          self.memory = []
          self.count = 0
  
      def push(self, experience):
          if len(self.memory) &lt; self.capacity:
              self.memory.append(experience)
          else:
              self.memory[self.count % self.capacity] = experience
          self.count += 1
  
      def sample(self, batch_size):
          return random.sample(self.memory, batch_size)
  
      def can_provide_sample(self, batch_size):
          return len(self.memory) &gt;= batch_size
  </code></pre>
  
  <p>This basically stores states/rewards/actions and returns a random group when asked.</p>
  
  <p>Lastly, here is my learning code:</p>
  
  <pre><code>def learning(num_episodes=20):
          global scores, experiences, target_vecs
          y = 0.8
          eps = 0
          decay_factor = 0.9999
  
  for i in range(num_episodes):
      state = GAME.reset()
      GAME.done = False
      done = False
      counter = 0
      while not done:
          eps *= decay_factor
          counter+=1
  
          if np.random.random() &lt; eps:
              a = np.random.randint(0, 2)
          else:
              a = np.argmax(model.predict(np.array([scale(state)])))
  
          new_state, reward, done = GAME.step(a) #does that step
          REPLAY_MEMORY.push((scale(state), a, reward, scale(new_state)))
  
          #experience replay is here
          if REPLAY_MEMORY.can_provide_sample(20):
              experiences = REPLAY_MEMORY.sample(20)
              target_vecs = []
              for j in range(len(experiences)):
                  target = experiences[j][2] + y * np.max(model.predict(np.array([experiences[j][3]])))
                  target_vec = model.predict(np.array([experiences[j][0]]))[0]
                  target_vec[experiences[j][1]] = target
                  target_vecs.append(target_vec)
              target_vecs = np.array(target_vecs)
              states = [s for s, _, _, _ in [exp for exp in experiences]]
              states = np.array(states)
              model.fit(states, target_vecs, epochs=1, verbose=1 if counter % 100 == 0 else 0)
          state = new_state
          if counter &gt; 1200: #game runs for 20 seconds each episode
              done = True
              scores.append(GAME.PLAYER.score)
  model.save("model.h5")
  </code></pre>
  
  <p>First, this takes a long time to train on my GTX1050. Is this normal for such a simple game? Also, does my code look fine? This is my first time with Deep Q Learning, so I would appreciate a second set of eyes. </p>
  
  <p>What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern.</p>
  
  <p><a href="https://repl.it/repls/DarkorchidTurbulentPolygon" rel="nofollow noreferrer">Here is a link to the code.</a> Also, available on <a href="https://github.com/msolonko/first_ai_game" rel="nofollow noreferrer">GitHub</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-06T20:14:11.863Z
Comments.Text: @NeilSlater Yes. Would you like me to just post the code or is there a better way to share it?
Comments.UserDisplayName: shurup
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T12:59:50.033Z
Comments.Text: To run the game using the model, I first run `model.save("model.h5").`. The run the game. I create a game object `g=Game()`. Then, I run `g.run(load_model("model.h5"))`. Let me know if this helps @NeilSlater
Comments.UserDisplayName: shurup
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T21:41:49.680Z
Comments.Text: @NeilSlater Also, when training, I don't actually run the game visually. Just using the `step` function that does all the math behind it but not the drawing. I also fixed an error where the inputs were not scaled when making predictions (when testing mode). The new code is on Github.
Comments.UserDisplayName: shurup
Tags: <deep-learning><game-ai><q-learning><keras><dqn>
Title: Deep Q Learning for Simple Game Not Effective
UserDisplayName: shurup
ThreadText: # Deep Q Learning for Simple Game Not Effective
  
  (Asked by: username_0 on 2019-08-06)
  
  This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (gain points) and some is bad (lose points). NN Architecture:
      #inputs - paddle.x, food.x, food.y, food.type
      #moves: left, right, stay
      model = Sequential()
      model.add(Dense(10, input_shape=(4,), activation='relu'))
      model.add(Dense(10, activation='relu'))
      model.add(Dense(3, activation='linear'))
      model.compile(loss='mean_squared_error', optimizer='adam')
  
  As suggested in the other question, I scaled my inputs to be between 0 and 1. Also, implemented experience replay (although I am not confident I did it correctly). 
  Here is my ReplayMemory class:
  class ReplayMemory():
      def __init__(self, capacity):
          self.capacity = capacity
          self.memory = []
          self.count = 0
  
      def push(self, experience):
          if len(self.memory) < self.capacity:
              self.memory.append(experience)
          else:
              self.memory[self.count % self.capacity] = experience
          self.count += 1
  
      def sample(self, batch_size):
          return random.sample(self.memory, batch_size)
  
      def can_provide_sample(self, batch_size):
          return len(self.memory) >= batch_size
  
  This basically stores states/rewards/actions and returns a random group when asked.
  Lastly, here is my learning code:
  def learning(num_episodes=20):
          global scores, experiences, target_vecs
          y = 0.8
          eps = 0
          decay_factor = 0.9999
  
  for i in range(num_episodes):
      state = GAME.reset()
      GAME.done = False
      done = False
      counter = 0
      while not done:
          eps *= decay_factor
          counter+=1
  
          if np.random.random() < eps:
              a = np.random.randint(0, 2)
          else:
              a = np.argmax(model.predict(np.array([scale(state)])))
  
          new_state, reward, done = GAME.step(a) #does that step
          REPLAY_MEMORY.push((scale(state), a, reward, scale(new_state)))
  
          #experience replay is here
          if REPLAY_MEMORY.can_provide_sample(20):
              experiences = REPLAY_MEMORY.sample(20)
              target_vecs = []
              for j in range(len(experiences)):
                  target = experiences[j][2] + y * np.max(model.predict(np.array([experiences[j][3]])))
                  target_vec = model.predict(np.array([experiences[j][0]]))[0]
                  target_vec[experiences[j][1]] = target
                  target_vecs.append(target_vec)
              target_vecs = np.array(target_vecs)
              states = [s for s, _, _, _ in [exp for exp in experiences]]
              states = np.array(states)
              model.fit(states, target_vecs, epochs=1, verbose=1 if counter % 100 == 0 else 0)
          state = new_state
          if counter > 1200: #game runs for 20 seconds each episode
              done = True
              scores.append(GAME.PLAYER.score)
  model.save("model.h5")
  
  First, this takes a long time to train on my GTX1050. Is this normal for such a simple game? Also, does my code look fine? This is my first time with Deep Q Learning, so I would appreciate a second set of eyes. 
  What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern.
  Here is a link to the code. Also, available on GitHub.
  
  **Tags:** deep-learning, game-ai, q-learning, keras, dqn
  > **Comments:**
  > **username_0**: @NeilSlater Yes. Would you like me to just post the code or is there a better way to share it?  
  > **username_0**: To run the game using the model, I first run `model.save("model.h5").`. The run the game. I create a game object `g=Game()`. Then, I run `g.run(load_model("model.h5"))`. Let me know if this helps @NeilSlater  
  > **username_0**: @NeilSlater Also, when training, I don't actually run the game visually. Just using the `step` function that does all the math behind it but not the drawing. I also fixed an error where the inputs were not scaled when making predictions (when testing mode). The new code is on Github.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-10. Score: 2
  
  Your implementation of single-step Q-learning with neural network and experience replay is basically correct.
  There are a few blocking issues preventing you seeing it working correctly.
  Your main problem is a bug in your feature scaling routine. That is a Python issue, not really an AI one. In short, you scale the input features in-place multiple times, including an effective double-scaling of next_state (when it gets copied to state you scale it in place a second time in the next loop) so that all the states that you store in the experience replay table never match to any input states. You need to change your definition of scale to not do this. A very simple re-write of your routine would be:
  def scale(s):
      return [s[0]/500, s[1]/500, s[2]/300, s[3]/3]
  
  In addition, you need to change random action selection to:
  np.random.randint(0, 3)
  
  because the end of range is never output (this matches behaviour of other range values and operators in Python). Not including the "do nothing" action during exploration means that the agent will test it less, and have less data to work with to assess whether it is the best action. This is a minor issue for this environment, but you should fix it nonetheless.
  
  What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). 
  
  I cannot replicate this fault and can train 20 episodes in around 2.5 minutes - that's over 20 times faster than you report. I am not using a GPU. Possibly in your case, Theano and Pygame are fighting for control of the GPU, or you may have a GPU configuration issue with Theano. Try turning GPU acceleraton off to verify whether it helps. You don't benefit much from a GPU for this environment (most time is spent in Python running the Q-learning and the environment), so can afford to put solving that issue to one side for now.
  
  Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern
  
  Sadly, I cannot see the output at all on my MacBook pro, but I was able to use feedback of the expected score. A random agent gets a mean score of ~5.5 per episode. With the scale function corrected, and a rough guess at working hyperparameters, I can get an average score of ~17 per episode consistently after 60 episodes of training. After 150 episodes - taking 20 minutes to train - the agent was scoring ~20 per episode and I stopped there. It is possible that an expected score around 20 is already optimal, as it is a very simple environment, but I don't know.
  Once you have a working system, there are lots of hyperparameters you could play with to try and improve this. I got my results by making the following changes after fixing the scale function:
  
  Starting epsilon of 1.0
  Repay memory size 10,000
  Only start learning when replay memory has greater than 1,000 entries
  Discount factor $\gamma$ 0.99
  Neural network with 20 neurons per layer with tanh activation instead of relu
  
  There is quite a lot else you could change that might make the agent learn more effectively or perhaps aim for a more optimal policy. Have fun experimenting!
  
  > **Comments:**
  > **username_0**: Thanks for taking the time to try my code! I implemented all that you said. I tried using `tensorflow` instead of `tensorflow-gpu` in order to just use the CPU. It is still just as slow. This loop: `for j in range(len(experiences)):` takes a lot of time I guess. Could this be an IDE issue? I am using Spyder. I also completely removed PyGame from the code (just while training).  
  > **username_1**: @NickSolonko: Sorry I cannot tell why you are experiencing such slow times. All I can say is that is not due to your code in general which runs fine on my MacBook (which is 2012 model, so not top of the range for a modern machine). You could vectorise that loop in Keras - if you are not sure how to, ask here or on SO, or on DataScience. Not sure if that will help you that much though, as the core problem seems to be somewhere else.  
  > **username_0**: Did it take your system 2.5 minutes even when sampling 1,000 Replay Memory records? I tried another Python IDE on another computer. It is a little faster, but will ask another question on Stack to try to fix the problem.  
  > **username_1**: @NickSolonko: I didn't try with just 1000, and cannot at the moment. I upgraded it to 10,000 for my test. I doubt my change would make it faster, as the 2.5 minutes was for 20 episodes (and 20 minutes total to reach top performance for the agent), and sampling from 10,000 entries would be slightly slower in Python, probably not noticeable though  
  
  ---
  
  