Body: <p>For my school project, I have to develop an agent to play my game.</p>
  <p><img src="https://i.sstatic.net/GEzsq.png" alt="The Board Game" /></p>
  <p>The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.</p>
  <p>To make my AI perform, I decided to make a deep RL algorithm.</p>
  <p>Here is how I've designed my solution.</p>
  <p>1st : the board is a 8x8 board. making <strong>112 possible lines to draw</strong>.
  2nd : on each decision, my Agent has to choose 1 line in the remaining one.
  3rd : each <strong>decision the Agent take is one among 112</strong> possible.</p>
  <p>I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.</p>
  <p>I made an architecture which is this one:
  a game is simulated:
  the board is clean, making all 112 possibilities available.
  Our Agent is interroged by the gameManager to make a move passing him the actual state of the game
  (<strong>the state shape is a 112*1 vector of Boolean values</strong>, 1 means a line can be drawn, 0 means there is already a line on this position)
  (<strong>the action shape is a vector of 112*1 Boolean values</strong>, All values are set to 'False' except the line we want to draw)
  So, our Agent return his move decision.</p>
  <p>Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.</p>
  <p>The rewards I choose are:
  +1 if our action make us close a box,
  -1 if our action make other close a box,
  +10 if our action make us win the game,
  -10 if our action make us loose the game</p>
  <p>The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing.
  when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.</p>
  <p>I give you the code I've wrote:</p>
  <p><strong>Here is the gameManager code:</strong></p>
  <pre><code>while True:
  hasMadeABox = False
  gameIsEnd = False
  rewardFCB = 0
  doneFCB = False
  
  
  if GRAPHIC_MODE:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              pygame.quit()
              sys.exit()
      disp_board()
  
  if playerTurns==&quot;1&quot;:
      stateFCB = possibilitiesToBoolList(possible_moves[0])
  
  boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])
  
  theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
  #print(possible_moves[0])
  #print(theAction)
  if playerTurns ==&quot;1&quot;:
      actionFCB = theAction
  
  if playerTurns==&quot;1&quot;:
      is_box = move(True, theAction)
  
  elif playerTurns==&quot;2&quot;:
      is_box = move(False, theAction)
  
  if is_box:
      if playerTurns ==&quot;1&quot;:
          #rewardFCB = 1
          rewardFCB = 1
          pass
      else:
          rewardFCB = -1
      hasMadeABox = True
  
  if check_complete():
      gameIsEnd = True
      rewardFCB += 10 if score[0]&gt;score[1] else -10 #does loosing is a reward null or negativ ?
      queueOfLastGame.pop(0)
  
      #Scotch pour affichage winrate
      isWin = 1 if score[0]&gt;score[1] else -1
      queueOfLastGame.append(isWin)
      if queueOfLastGame.count(-1)+queueOfLastGame.count(1) &gt; 0:
          print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , &quot; % Winrate&quot;)
  
      doneFCB = True
  
  
  if playerTurns==&quot;1&quot; and hasMadeABox:
      #si c'est notre IA vient de faire un carré
      #on connait directement l'état qui succede
      nextStateFCB = possibilitiesToBoolList(possible_moves[0])
  
  if playerTurns==&quot;2&quot;:
      nextStateFCB = possibilitiesToBoolList(possible_moves[0])
  
  
  if nextStateFCB is not None:
      bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
      #ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
      rewardFCB = 0
      nextStateFCB = None
  
  if gameIsEnd:
      flushBufferSARS()
      reset()
      continue
  
  #switch user to play if game is not end
  if not hasMadeABox:
      playerTurns=&quot;1&quot; if playerTurns == &quot;2&quot; else &quot;2&quot;
  </code></pre>
  <p><strong>And here's my code about the Agent:</strong></p>
  <pre><code>class Agent:
  def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):
  
      self._memory = deque(maxlen=100000)
      self._actualEpisode=1
      self._episodes=7000
      self._name=name
      self._possibleAction=possibleActions
      self._isHuman=isHuman
      self._epsilon=epsilon
      self._epsilonDecay = 0.99
      self._epsilonMin = 0.05
      self._gamma=gamma
      self._stateSize=stateSize
      self._actionSize=actionSize
      self._alpha=alpha
      self._alphaDecay=alphaDecay
      self._hasToTrain=hasToTrain
      self._batchSize=batchSize
  
      self._totalIllegalMove = 0
      self._totalLegalMove = 0
  
      self._path = &quot;./modelWeightSave/&quot;
  
      self._model = self._buildModel()
  
  
  def save_model(self):
      self._model.save(self._path)
  
  def getName(self):
      return self._name
  
  def _buildModel(self):
      model = Sequential()
  
      model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
      model.add(Dense(256, kernel_initializer='normal', activation='relu'))
      model.add(Dense(256, kernel_initializer='normal', activation='relu'))
      model.add(Dense(256, kernel_initializer='normal', activation='relu'))
      model.add(Dense(128, kernel_initializer='normal', activation='relu'))
      model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))
  
      model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
      #if os.path.isfile(self._path):
          #model.load_weights(self._path)
      return model
  
  def act(self, UNUSED_state, stateAsBool):
      playableIndexes = []
      for i in range(len(stateAsBool[0])):
          if stateAsBool[0][i] == 1:
              playableIndexes.append(i)
      indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]
  
      if np.random.random() &lt;= self._epsilon:
          action= [0]*self._actionSize
          action[indexForRand]=1
  
      else:
          arrayState = np.array(stateAsBool)
  
          action = self._model.predict(arrayState)
          #Set index of max esperence to 1, we play this line.
          tmp=[0]*self._actionSize
          tmp[np.argmax(action)] = 1
          action = tmp
  
          isLegalMove = True
          if sum(action) != 1:
              isLegalMove = False
          for i in range(len(action)):
              if action[i] == 1:
                  if stateAsBool[0][i] == 0:
                      isLegalMove = False
                      break
  
          if isLegalMove:
              pass
              #print(&quot;Legal move&quot;)
          else:
              #print(&quot;Illegal move&quot;)
              #AI try to play on an already draw line, we choose a random line in remainings
              self._totalIllegalMove+=1
              action = [0] * self._actionSize
              action[indexForRand] = 1
  
      #print(&quot;My AI took action : &quot;,action)
      return action
  
  def remember(self, state, action, reward, nextState, done):
      self._memory.append((state.copy(), action, reward, nextState, done))
  
      self._actualEpisode+=1
      if self._actualEpisode &gt; self._episodes:
          self._actualEpisode = 0
          self.replay(self._batchSize)
  
  def replay(self, batchSize):
      x_batch, y_batch = [], []
      minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
      for state, action, reward, next_state, done in minibatch:
          actionIndex = np.argmax(action)
          y_target = self._model.predict(state)
          y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
          x_batch.append(state[0])
          y_batch.append(y_target[0])
      self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)
  
      if self._epsilon &gt; self._epsilonMin:
          self._epsilon *= self._epsilonDecay
  
      self.save_model()
  
  def play(self, board, state, statesAsBool, player):
      actionTaken= self.act(state, statesAsBool)
      return actionTaken
  
  def callBackOnPreviousMove(self, state, action, reward, nextState, done):
      self.remember(state, action, reward, nextState, done)
  </code></pre>
  <p>Example of output i have during fit method:</p>
  <pre><code>Epoch 1/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
   
  Epoch 2/10 
  
  1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867 
  
  Epoch 3/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867 
  
  Epoch 4/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867 
  
  Epoch 5/10 
  
  1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867 
  
  Epoch 6/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867 
  
  Epoch 7/10 
  
  1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867 
  
  Epoch 8/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867 
  
  Epoch 9/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867 
  
  Epoch 10/10
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867
  </code></pre>
  <p><strong>My questions are:</strong></p>
  <ol>
  <li><p>Is my architecture good
  (inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and<br />
  output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') )
  to represent an action ?</p>
  </li>
  <li><p>How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)</p>
  </li>
  <li><p>To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?</p>
  </li>
  <li><p>Is there something really important I forget in my design to make it work?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-04T18:14:57.787Z
Comments.Text: Please, ask one question per post. If you have multiple questions, ask each of them in their separate post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-04T18:18:13.893Z
Comments.Text: they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks
Comments.UserDisplayName: Benjamin Darras
Tags: <machine-learning><ai-design><python><tensorflow><keras>
Title: How to design my Neural Network for Game AI
UserDisplayName: Benjamin Darras
ThreadText: # How to design my Neural Network for Game AI
  
  (Asked by: username_0 on 2020-11-03)
  
  For my school project, I have to develop an agent to play my game.
  
  The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.
  To make my AI perform, I decided to make a deep RL algorithm.
  Here is how I've designed my solution.
  1st : the board is a 8x8 board. making 112 possible lines to draw.
  2nd : on each decision, my Agent has to choose 1 line in the remaining one.
  3rd : each decision the Agent take is one among 112 possible.
  I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.
  I made an architecture which is this one:
  a game is simulated:
  the board is clean, making all 112 possibilities available.
  Our Agent is interroged by the gameManager to make a move passing him the actual state of the game
  (the state shape is a 112*1 vector of Boolean values, 1 means a line can be drawn, 0 means there is already a line on this position)
  (the action shape is a vector of 112*1 Boolean values, All values are set to 'False' except the line we want to draw)
  So, our Agent return his move decision.
  Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.
  The rewards I choose are:
  +1 if our action make us close a box,
  -1 if our action make other close a box,
  +10 if our action make us win the game,
  -10 if our action make us loose the game
  The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing.
  when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.
  I give you the code I've wrote:
  Here is the gameManager code:
  while True:
  hasMadeABox = False
  gameIsEnd = False
  rewardFCB = 0
  doneFCB = False
  
  
  if GRAPHIC_MODE:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              pygame.quit()
              sys.exit()
      disp_board()
  
  if playerTurns=="1":
      stateFCB = possibilitiesToBoolList(possible_moves[0])
  
  boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])
  
  theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
  #print(possible_moves[0])
  #print(theAction)
  if playerTurns =="1":
      actionFCB = theAction
  
  if playerTurns=="1":
      is_box = move(True, theAction)
  
  elif playerTurns=="2":
      is_box = move(False, theAction)
  
  if is_box:
      if playerTurns =="1":
          #rewardFCB = 1
          rewardFCB = 1
          pass
      else:
          rewardFCB = -1
      hasMadeABox = True
  
  if check_complete():
      gameIsEnd = True
      rewardFCB += 10 if score[0]>score[1] else -10 #does loosing is a reward null or negativ ?
      queueOfLastGame.pop(0)
  
      #Scotch pour affichage winrate
      isWin = 1 if score[0]>score[1] else -1
      queueOfLastGame.append(isWin)
      if queueOfLastGame.count(-1)+queueOfLastGame.count(1) > 0:
          print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , " % Winrate")
  
      doneFCB = True
  
  
  if playerTurns=="1" and hasMadeABox:
      #si c'est notre IA vient de faire un carré
      #on connait directement l'état qui succede
      nextStateFCB = possibilitiesToBoolList(possible_moves[0])
  
  if playerTurns=="2":
      nextStateFCB = possibilitiesToBoolList(possible_moves[0])
  
  
  if nextStateFCB is not None:
      bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
      #ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
      rewardFCB = 0
      nextStateFCB = None
  
  if gameIsEnd:
      flushBufferSARS()
      reset()
      continue
  
  #switch user to play if game is not end
  if not hasMadeABox:
      playerTurns="1" if playerTurns == "2" else "2"
  
  And here's my code about the Agent:
  class Agent:
  def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):
  
      self._memory = deque(maxlen=100000)
      self._actualEpisode=1
      self._episodes=7000
      self._name=name
      self._possibleAction=possibleActions
      self._isHuman=isHuman
      self._epsilon=epsilon
      self._epsilonDecay = 0.99
      self._epsilonMin = 0.05
      self._gamma=gamma
      self._stateSize=stateSize
      self._actionSize=actionSize
      self._alpha=alpha
      self._alphaDecay=alphaDecay
      self._hasToTrain=hasToTrain
      self._batchSize=batchSize
  
      self._totalIllegalMove = 0
      self._totalLegalMove = 0
  
      self._path = "./modelWeightSave/"
  
      self._model = self._buildModel()
  
  
  def save_model(self):
      self._model.save(self._path)
  
  def getName(self):
      return self._name
  
  def _buildModel(self):
      model = Sequential()
  
      model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
      model.add(Dense(256, kernel_initializer='normal', activation='relu'))
      model.add(Dense(256, kernel_initializer='normal', activation='relu'))
      model.add(Dense(256, kernel_initializer='normal', activation='relu'))
      model.add(Dense(128, kernel_initializer='normal', activation='relu'))
      model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))
  
      model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
      #if os.path.isfile(self._path):
          #model.load_weights(self._path)
      return model
  
  def act(self, UNUSED_state, stateAsBool):
      playableIndexes = []
      for i in range(len(stateAsBool[0])):
          if stateAsBool[0][i] == 1:
              playableIndexes.append(i)
      indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]
  
      if np.random.random() <= self._epsilon:
          action= [0]*self._actionSize
          action[indexForRand]=1
  
      else:
          arrayState = np.array(stateAsBool)
  
          action = self._model.predict(arrayState)
          #Set index of max esperence to 1, we play this line.
          tmp=[0]*self._actionSize
          tmp[np.argmax(action)] = 1
          action = tmp
  
          isLegalMove = True
          if sum(action) != 1:
              isLegalMove = False
          for i in range(len(action)):
              if action[i] == 1:
                  if stateAsBool[0][i] == 0:
                      isLegalMove = False
                      break
  
          if isLegalMove:
              pass
              #print("Legal move")
          else:
              #print("Illegal move")
              #AI try to play on an already draw line, we choose a random line in remainings
              self._totalIllegalMove+=1
              action = [0] * self._actionSize
              action[indexForRand] = 1
  
      #print("My AI took action : ",action)
      return action
  
  def remember(self, state, action, reward, nextState, done):
      self._memory.append((state.copy(), action, reward, nextState, done))
  
      self._actualEpisode+=1
      if self._actualEpisode > self._episodes:
          self._actualEpisode = 0
          self.replay(self._batchSize)
  
  def replay(self, batchSize):
      x_batch, y_batch = [], []
      minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
      for state, action, reward, next_state, done in minibatch:
          actionIndex = np.argmax(action)
          y_target = self._model.predict(state)
          y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
          x_batch.append(state[0])
          y_batch.append(y_target[0])
      self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)
  
      if self._epsilon > self._epsilonMin:
          self._epsilon *= self._epsilonDecay
  
      self.save_model()
  
  def play(self, board, state, statesAsBool, player):
      actionTaken= self.act(state, statesAsBool)
      return actionTaken
  
  def callBackOnPreviousMove(self, state, action, reward, nextState, done):
      self.remember(state, action, reward, nextState, done)
  
  Example of output i have during fit method:
  Epoch 1/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
   
  Epoch 2/10 
  
  1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867 
  
  Epoch 3/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867 
  
  Epoch 4/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867 
  
  Epoch 5/10 
  
  1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867 
  
  Epoch 6/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867 
  
  Epoch 7/10 
  
  1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867 
  
  Epoch 8/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867 
  
  Epoch 9/10 
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867 
  
  Epoch 10/10
  
  1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867
  
  My questions are:
  
  Is my architecture good
  (inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and
  output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') )
  to represent an action ?
  
  How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)
  
  To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?
  
  Is there something really important I forget in my design to make it work?
  
  **Tags:** machine-learning, ai-design, python, tensorflow, keras
  > **Comments:**
  > **username_1**: Please, ask one question per post. If you have multiple questions, ask each of them in their separate post.  
  > **username_0**: they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks  
  
  ---
  
  ## 0 answers
  
  