Body: <p>I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game <a href="https://play2048.co/" rel="nofollow noreferrer">2048</a>. I am orientating myself on other programs and articles that are based on the game snake and it worked well (<a href="https://towardsdatascience.com/how-to-teach-an-ai-to-play-games-deep-reinforcement-learning-28f9b920440a" rel="nofollow noreferrer">specifically this one</a>).</p>
  
  <p>As input state, I am only using the grid with the tiles as numpy array, and as a reward, I use (newScore-oldScore-1) to penalize moves that do not give any points at all. I know that this might not be optimal, as one might as well reward staying alive for as long as possible, but it should be okay for the first step, right? Nevertheless, I am not getting any good results whatsoever.</p>
  
  <p>I've tried to tweak the model layout, the number of neurons and layers, optimizer, gamma, learning rates, rewards, <em>etc.</em>. I also tried ending the game after 5 moves and to optimize just for those first five moves but no matter what I do, I don't get any noticeable improvement. I've run it for thousands of games and it just doesn't get better. In fact, sometimes I get worse results than a completely random algorithm, as sometimes it just returns the same output for any input and gets stuck.</p>
  
  <p>So, my question is, if I am doing anything fundamentally wrong? Do I just have a small stupid mistake somewhere? Is this the wrong approach completely? (I know the game could probably be solved pretty easily without AI, but it seemed like a little fun project) </p>
  
  <p>My Jupyter notebook can be seen here <a href="https://github.com/Drukob/2048DQN" rel="nofollow noreferrer">Github</a>. Sorry for the poor code quality, I'm still a beginner and I know I need to start making documentation even for fun little projects...</p>
  
  <p>Thank you in advance,</p>
  
  <p>Drukob</p>
  
  <p>edit:
  some code snippets:</p>
  
  <p>Input is formatted as a 1,16 numpy array, also tried normalizing the values or using only 1 and 0 for occupied and empty cells, but that did not help either. Which is why I assumed it's maybe more of a conceptual problem?</p>
  
  
  
  <pre><code>    def get_board(self):
          grid = self.driver.execute_script("return myGM.grid.cells;")
          mygrid = []
          for line in grid:
              a = [x['value'] if x != None else 0 for x in line]
              #a = [1 if x != None else 0 for x in line]
              mygrid.append(a)
          return np.array(mygrid).reshape(1,16)
  </code></pre>
  
  <p>The output is an index of {0,3}, representing the actions up, down, left or right and it's just the value with the highes prediction score.</p>
  
  
  
  <pre><code>prediction = agent.model.predict(old_state)
  predicted_move = np.argmax(prediction)
  </code></pre>
  
  <p>I've tried a lot of different model architectures, but settled for a simpler network now, as I have read that unnecessary complex structures are often a problem and unneeded. However, I couldn't find any reliable source for a method, how to get the optimal layout except for experimenting, so I'd be happy to have some more suggestions there.</p>
  
  
  
  <pre><code>model = models.Sequential()
          model.add(Dense(16, activation='relu', input_dim=16))
          #model.add(Dropout(0.15))
          #model.add(Dense(50, activation='relu'))
          #model.add(Dropout(0.15))
          model.add(Dense(20, activation='relu'))
          #model.add(Dropout(0.15))
          #model.add(Dense(30, input_dim=16, activation='relu'))
          #model.add(Dropout(0.15))
          #model.add(Dense(30, activation='relu'))
          #model.add(Dropout(0.15))
          #model.add(Dense(8, activation='relu'))
          #model.add(Dropout(0.15))
          model.add(Dense(4, activation='linear'))
          opt = Adam(lr=self.learning_rate)
          model.compile(loss='mse', optimizer=opt)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-13T06:58:17.167Z
Comments.Text: Could you give details in the question, how you are representing the input? Are scaling it in any way? Please give *small* snippet of relevant code if that makes it clear (don't expect anyone will read your whole notebook just to find problems for you). use [edit] to add details. Another common mistake could be in how you are representing the output - what is your output layer, is it multi-action or single action, have you added any non-linearity?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-17T12:26:01.737Z
Comments.Text: Could you explain what you mean with multi-action or single-action? My output is just a 1-D Vector with four values for the four possible actions (up, down, left, right) of which the highest value will be taken and this action is then performed.
Comments.UserDisplayName: Drukob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-17T14:41:06.220Z
Comments.Text: That's what I'd call multi-action. The NN outputs a value for each possible action, given a state, all at once. The alternative would be to model $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ directly, with input of state and action representation concatenated, and output a single value.
Comments.UserDisplayName: Neil Slater
Tags: <python><game-ai><dqn><deep-rl>
Title: Deep Q-Network (DQN) to learn the game 2048
UserDisplayName: Drukob
ThreadText: # Deep Q-Network (DQN) to learn the game 2048
  
  (Asked by: username_0 on 2019-06-12)
  
  I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game 2048. I am orientating myself on other programs and articles that are based on the game snake and it worked well (specifically this one).
  As input state, I am only using the grid with the tiles as numpy array, and as a reward, I use (newScore-oldScore-1) to penalize moves that do not give any points at all. I know that this might not be optimal, as one might as well reward staying alive for as long as possible, but it should be okay for the first step, right? Nevertheless, I am not getting any good results whatsoever.
  I've tried to tweak the model layout, the number of neurons and layers, optimizer, gamma, learning rates, rewards, etc.. I also tried ending the game after 5 moves and to optimize just for those first five moves but no matter what I do, I don't get any noticeable improvement. I've run it for thousands of games and it just doesn't get better. In fact, sometimes I get worse results than a completely random algorithm, as sometimes it just returns the same output for any input and gets stuck.
  So, my question is, if I am doing anything fundamentally wrong? Do I just have a small stupid mistake somewhere? Is this the wrong approach completely? (I know the game could probably be solved pretty easily without AI, but it seemed like a little fun project) 
  My Jupyter notebook can be seen here Github. Sorry for the poor code quality, I'm still a beginner and I know I need to start making documentation even for fun little projects...
  Thank you in advance,
  Drukob
  edit:
  some code snippets:
  Input is formatted as a 1,16 numpy array, also tried normalizing the values or using only 1 and 0 for occupied and empty cells, but that did not help either. Which is why I assumed it's maybe more of a conceptual problem?
      def get_board(self):
          grid = self.driver.execute_script("return myGM.grid.cells;")
          mygrid = []
          for line in grid:
              a = [x['value'] if x != None else 0 for x in line]
              #a = [1 if x != None else 0 for x in line]
              mygrid.append(a)
          return np.array(mygrid).reshape(1,16)
  
  The output is an index of {0,3}, representing the actions up, down, left or right and it's just the value with the highes prediction score.
  prediction = agent.model.predict(old_state)
  predicted_move = np.argmax(prediction)
  
  I've tried a lot of different model architectures, but settled for a simpler network now, as I have read that unnecessary complex structures are often a problem and unneeded. However, I couldn't find any reliable source for a method, how to get the optimal layout except for experimenting, so I'd be happy to have some more suggestions there.
  model = models.Sequential()
          model.add(Dense(16, activation='relu', input_dim=16))
          #model.add(Dropout(0.15))
          #model.add(Dense(50, activation='relu'))
          #model.add(Dropout(0.15))
          model.add(Dense(20, activation='relu'))
          #model.add(Dropout(0.15))
          #model.add(Dense(30, input_dim=16, activation='relu'))
          #model.add(Dropout(0.15))
          #model.add(Dense(30, activation='relu'))
          #model.add(Dropout(0.15))
          #model.add(Dense(8, activation='relu'))
          #model.add(Dropout(0.15))
          model.add(Dense(4, activation='linear'))
          opt = Adam(lr=self.learning_rate)
          model.compile(loss='mse', optimizer=opt)
  
  **Tags:** python, game-ai, dqn, deep-rl
  > **Comments:**
  > **username_1**: Could you give details in the question, how you are representing the input? Are scaling it in any way? Please give *small* snippet of relevant code if that makes it clear (don't expect anyone will read your whole notebook just to find problems for you). use [edit] to add details. Another common mistake could be in how you are representing the output - what is your output layer, is it multi-action or single action, have you added any non-linearity?  
  > **username_0**: Could you explain what you mean with multi-action or single-action? My output is just a 1-D Vector with four values for the four possible actions (up, down, left, right) of which the highest value will be taken and this action is then performed.  
  > **username_1**: That's what I'd call multi-action. The NN outputs a value for each possible action, given a state, all at once. The alternative would be to model $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ directly, with input of state and action representation concatenated, and output a single value.  
  
  ---
  
  ## 0 answers
  
  