<p>For my school project, I have to develop an agent to play my game.</p>
<p><img src="https://i.sstatic.net/GEzsq.png" alt="The Board Game" /></p>
<p>The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.</p>
<p>To make my AI perform, I decided to make a deep RL algorithm.</p>
<p>Here is how I've designed my solution.</p>
<p>1st : the board is a 8x8 board. making <strong>112 possible lines to draw</strong>.
2nd : on each decision, my Agent has to choose 1 line in the remaining one.
3rd : each <strong>decision the Agent take is one among 112</strong> possible.</p>
<p>I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.</p>
<p>I made an architecture which is this one:
a game is simulated:
the board is clean, making all 112 possibilities available.
Our Agent is interroged by the gameManager to make a move passing him the actual state of the game
(<strong>the state shape is a 112*1 vector of Boolean values</strong>, 1 means a line can be drawn, 0 means there is already a line on this position)
(<strong>the action shape is a vector of 112*1 Boolean values</strong>, All values are set to 'False' except the line we want to draw)
So, our Agent return his move decision.</p>
<p>Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.</p>
<p>The rewards I choose are:
+1 if our action make us close a box,
-1 if our action make other close a box,
+10 if our action make us win the game,
-10 if our action make us loose the game</p>
<p>The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing.
when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.</p>
<p>I give you the code I've wrote:</p>
<p><strong>Here is the gameManager code:</strong></p>
<pre><code>while True:
hasMadeABox = False
gameIsEnd = False
rewardFCB = 0
doneFCB = False


if GRAPHIC_MODE:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            sys.exit()
    disp_board()

if playerTurns==&quot;1&quot;:
    stateFCB = possibilitiesToBoolList(possible_moves[0])

boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])

theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
#print(possible_moves[0])
#print(theAction)
if playerTurns ==&quot;1&quot;:
    actionFCB = theAction

if playerTurns==&quot;1&quot;:
    is_box = move(True, theAction)

elif playerTurns==&quot;2&quot;:
    is_box = move(False, theAction)

if is_box:
    if playerTurns ==&quot;1&quot;:
        #rewardFCB = 1
        rewardFCB = 1
        pass
    else:
        rewardFCB = -1
    hasMadeABox = True

if check_complete():
    gameIsEnd = True
    rewardFCB += 10 if score[0]&gt;score[1] else -10 #does loosing is a reward null or negativ ?
    queueOfLastGame.pop(0)

    #Scotch pour affichage winrate
    isWin = 1 if score[0]&gt;score[1] else -1
    queueOfLastGame.append(isWin)
    if queueOfLastGame.count(-1)+queueOfLastGame.count(1) &gt; 0:
        print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , &quot; % Winrate&quot;)

    doneFCB = True


if playerTurns==&quot;1&quot; and hasMadeABox:
    #si c'est notre IA vient de faire un carré
    #on connait directement l'état qui succede
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if playerTurns==&quot;2&quot;:
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])


if nextStateFCB is not None:
    bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
    #ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
    rewardFCB = 0
    nextStateFCB = None

if gameIsEnd:
    flushBufferSARS()
    reset()
    continue

#switch user to play if game is not end
if not hasMadeABox:
    playerTurns=&quot;1&quot; if playerTurns == &quot;2&quot; else &quot;2&quot;
</code></pre>
<p><strong>And here's my code about the Agent:</strong></p>
<pre><code>class Agent:
def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):

    self._memory = deque(maxlen=100000)
    self._actualEpisode=1
    self._episodes=7000
    self._name=name
    self._possibleAction=possibleActions
    self._isHuman=isHuman
    self._epsilon=epsilon
    self._epsilonDecay = 0.99
    self._epsilonMin = 0.05
    self._gamma=gamma
    self._stateSize=stateSize
    self._actionSize=actionSize
    self._alpha=alpha
    self._alphaDecay=alphaDecay
    self._hasToTrain=hasToTrain
    self._batchSize=batchSize

    self._totalIllegalMove = 0
    self._totalLegalMove = 0

    self._path = &quot;./modelWeightSave/&quot;

    self._model = self._buildModel()


def save_model(self):
    self._model.save(self._path)

def getName(self):
    return self._name

def _buildModel(self):
    model = Sequential()

    model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))

    model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
    #if os.path.isfile(self._path):
        #model.load_weights(self._path)
    return model

def act(self, UNUSED_state, stateAsBool):
    playableIndexes = []
    for i in range(len(stateAsBool[0])):
        if stateAsBool[0][i] == 1:
            playableIndexes.append(i)
    indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]

    if np.random.random() &lt;= self._epsilon:
        action= [0]*self._actionSize
        action[indexForRand]=1

    else:
        arrayState = np.array(stateAsBool)

        action = self._model.predict(arrayState)
        #Set index of max esperence to 1, we play this line.
        tmp=[0]*self._actionSize
        tmp[np.argmax(action)] = 1
        action = tmp

        isLegalMove = True
        if sum(action) != 1:
            isLegalMove = False
        for i in range(len(action)):
            if action[i] == 1:
                if stateAsBool[0][i] == 0:
                    isLegalMove = False
                    break

        if isLegalMove:
            pass
            #print(&quot;Legal move&quot;)
        else:
            #print(&quot;Illegal move&quot;)
            #AI try to play on an already draw line, we choose a random line in remainings
            self._totalIllegalMove+=1
            action = [0] * self._actionSize
            action[indexForRand] = 1

    #print(&quot;My AI took action : &quot;,action)
    return action

def remember(self, state, action, reward, nextState, done):
    self._memory.append((state.copy(), action, reward, nextState, done))

    self._actualEpisode+=1
    if self._actualEpisode &gt; self._episodes:
        self._actualEpisode = 0
        self.replay(self._batchSize)

def replay(self, batchSize):
    x_batch, y_batch = [], []
    minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
    for state, action, reward, next_state, done in minibatch:
        actionIndex = np.argmax(action)
        y_target = self._model.predict(state)
        y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
        x_batch.append(state[0])
        y_batch.append(y_target[0])
    self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)

    if self._epsilon &gt; self._epsilonMin:
        self._epsilon *= self._epsilonDecay

    self.save_model()

def play(self, board, state, statesAsBool, player):
    actionTaken= self.act(state, statesAsBool)
    return actionTaken

def callBackOnPreviousMove(self, state, action, reward, nextState, done):
    self.remember(state, action, reward, nextState, done)
</code></pre>
<p>Example of output i have during fit method:</p>
<pre><code>Epoch 1/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
 
Epoch 2/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867 

Epoch 3/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867 

Epoch 4/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867 

Epoch 5/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867 

Epoch 6/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867 

Epoch 7/10 

1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867 

Epoch 8/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867 

Epoch 9/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867 

Epoch 10/10

1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867
</code></pre>
<p><strong>My questions are:</strong></p>
<ol>
<li><p>Is my architecture good
(inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and<br />
output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') )
to represent an action ?</p>
</li>
<li><p>How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)</p>
</li>
<li><p>To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?</p>
</li>
<li><p>Is there something really important I forget in my design to make it work?</p>
</li>
</ol>

CC BY-SA 4.0
2020-11-04T18:14:57.787Z
Please, ask one question per post. If you have multiple questions, ask each of them in their separate post.
nbro
CC BY-SA 4.0
2020-11-04T18:18:13.893Z
they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks
Benjamin Darras
<machine-learning><ai-design><python><tensorflow><keras>
How to design my Neural Network for Game AI
Benjamin Darras
# How to design my Neural Network for Game AI

(Asked by: username_0 on 2020-11-03)

For my school project, I have to develop an agent to play my game.

The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.
To make my AI perform, I decided to make a deep RL algorithm.
Here is how I've designed my solution.
1st : the board is a 8x8 board. making 112 possible lines to draw.
2nd : on each decision, my Agent has to choose 1 line in the remaining one.
3rd : each decision the Agent take is one among 112 possible.
I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.
I made an architecture which is this one:
a game is simulated:
the board is clean, making all 112 possibilities available.
Our Agent is interroged by the gameManager to make a move passing him the actual state of the game
(the state shape is a 112*1 vector of Boolean values, 1 means a line can be drawn, 0 means there is already a line on this position)
(the action shape is a vector of 112*1 Boolean values, All values are set to 'False' except the line we want to draw)
So, our Agent return his move decision.
Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.
The rewards I choose are:
+1 if our action make us close a box,
-1 if our action make other close a box,
+10 if our action make us win the game,
-10 if our action make us loose the game
The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing.
when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.
I give you the code I've wrote:
Here is the gameManager code:
while True:
hasMadeABox = False
gameIsEnd = False
rewardFCB = 0
doneFCB = False


if GRAPHIC_MODE:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            sys.exit()
    disp_board()

if playerTurns=="1":
    stateFCB = possibilitiesToBoolList(possible_moves[0])

boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])

theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
#print(possible_moves[0])
#print(theAction)
if playerTurns =="1":
    actionFCB = theAction

if playerTurns=="1":
    is_box = move(True, theAction)

elif playerTurns=="2":
    is_box = move(False, theAction)

if is_box:
    if playerTurns =="1":
        #rewardFCB = 1
        rewardFCB = 1
        pass
    else:
        rewardFCB = -1
    hasMadeABox = True

if check_complete():
    gameIsEnd = True
    rewardFCB += 10 if score[0]>score[1] else -10 #does loosing is a reward null or negativ ?
    queueOfLastGame.pop(0)

    #Scotch pour affichage winrate
    isWin = 1 if score[0]>score[1] else -1
    queueOfLastGame.append(isWin)
    if queueOfLastGame.count(-1)+queueOfLastGame.count(1) > 0:
        print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , " % Winrate")

    doneFCB = True


if playerTurns=="1" and hasMadeABox:
    #si c'est notre IA vient de faire un carré
    #on connait directement l'état qui succede
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if playerTurns=="2":
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])


if nextStateFCB is not None:
    bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
    #ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
    rewardFCB = 0
    nextStateFCB = None

if gameIsEnd:
    flushBufferSARS()
    reset()
    continue

#switch user to play if game is not end
if not hasMadeABox:
    playerTurns="1" if playerTurns == "2" else "2"

And here's my code about the Agent:
class Agent:
def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):

    self._memory = deque(maxlen=100000)
    self._actualEpisode=1
    self._episodes=7000
    self._name=name
    self._possibleAction=possibleActions
    self._isHuman=isHuman
    self._epsilon=epsilon
    self._epsilonDecay = 0.99
    self._epsilonMin = 0.05
    self._gamma=gamma
    self._stateSize=stateSize
    self._actionSize=actionSize
    self._alpha=alpha
    self._alphaDecay=alphaDecay
    self._hasToTrain=hasToTrain
    self._batchSize=batchSize

    self._totalIllegalMove = 0
    self._totalLegalMove = 0

    self._path = "./modelWeightSave/"

    self._model = self._buildModel()


def save_model(self):
    self._model.save(self._path)

def getName(self):
    return self._name

def _buildModel(self):
    model = Sequential()

    model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))

    model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
    #if os.path.isfile(self._path):
        #model.load_weights(self._path)
    return model

def act(self, UNUSED_state, stateAsBool):
    playableIndexes = []
    for i in range(len(stateAsBool[0])):
        if stateAsBool[0][i] == 1:
            playableIndexes.append(i)
    indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]

    if np.random.random() <= self._epsilon:
        action= [0]*self._actionSize
        action[indexForRand]=1

    else:
        arrayState = np.array(stateAsBool)

        action = self._model.predict(arrayState)
        #Set index of max esperence to 1, we play this line.
        tmp=[0]*self._actionSize
        tmp[np.argmax(action)] = 1
        action = tmp

        isLegalMove = True
        if sum(action) != 1:
            isLegalMove = False
        for i in range(len(action)):
            if action[i] == 1:
                if stateAsBool[0][i] == 0:
                    isLegalMove = False
                    break

        if isLegalMove:
            pass
            #print("Legal move")
        else:
            #print("Illegal move")
            #AI try to play on an already draw line, we choose a random line in remainings
            self._totalIllegalMove+=1
            action = [0] * self._actionSize
            action[indexForRand] = 1

    #print("My AI took action : ",action)
    return action

def remember(self, state, action, reward, nextState, done):
    self._memory.append((state.copy(), action, reward, nextState, done))

    self._actualEpisode+=1
    if self._actualEpisode > self._episodes:
        self._actualEpisode = 0
        self.replay(self._batchSize)

def replay(self, batchSize):
    x_batch, y_batch = [], []
    minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
    for state, action, reward, next_state, done in minibatch:
        actionIndex = np.argmax(action)
        y_target = self._model.predict(state)
        y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
        x_batch.append(state[0])
        y_batch.append(y_target[0])
    self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)

    if self._epsilon > self._epsilonMin:
        self._epsilon *= self._epsilonDecay

    self.save_model()

def play(self, board, state, statesAsBool, player):
    actionTaken= self.act(state, statesAsBool)
    return actionTaken

def callBackOnPreviousMove(self, state, action, reward, nextState, done):
    self.remember(state, action, reward, nextState, done)

Example of output i have during fit method:
Epoch 1/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
 
Epoch 2/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867 

Epoch 3/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867 

Epoch 4/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867 

Epoch 5/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867 

Epoch 6/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867 

Epoch 7/10 

1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867 

Epoch 8/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867 

Epoch 9/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867 

Epoch 10/10

1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867

My questions are:

Is my architecture good
(inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and
output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') )
to represent an action ?

How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)

To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?

Is there something really important I forget in my design to make it work?

**Tags:** machine-learning, ai-design, python, tensorflow, keras

> **Comments:**
> **username_1**: Please, ask one question per post. If you have multiple questions, ask each of them in their separate post.  
> **username_0**: they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks  

---

## 0 answers

