Body: <p>I'm attempting to make a bot for the Connect 4 competition on <a href="http://riddles.io" rel="nofollow noreferrer">http://riddles.io</a></p>
  
  <p>My bot isn't horrible, like it's getting up the ladder, but it cannot compete with the winning bots.</p>
  
  <p>I'm using a Neural Network which is fully connected with one hidden layer. Internally it uses the sigmoid function as the activator in each Neuron. I've trained it over 500,000 games with TD-Lambda back propagation. The alpha and beta values (i.e. learning rates) are set to <code>0.1</code> each, and the lambda for the eligibility trace is set to <code>0.7</code>. There are 2 outputs nodes, one to give the value of this position for Player 1, the other to give the value of the position for Player 2. Upon a win, these are back-propagated with <code>-1</code> for a loss and <code>1</code> for a win, on a draw they are both back-propagated with a <code>0</code>.</p>
  
  <p>There is a bias input and weight for every neuron as well.</p>
  
  <p>All of the weights are initialised to a random value between <code>+-4*sqrt(6/totalNumberWeights)</code>.</p>
  
  <p>The board state is represented to the network as:</p>
  
  <ol>
  <li>For each space in the board, 2 values:
  
  <ul>
  <li>For the first value if player 1 occupies this space it's a <code>1</code>, otherwise it's a <code>0</code></li>
  <li>For the second value if player 2 occupies this space it's a <code>1</code>, otherwise it's a <code>0</code></li>
  <li>If both are <code>0</code> it would mean it's a free spot</li>
  </ul></li>
  <li>For each space on the board another 2 values:
  
  <ul>
  <li>For the first value, if placing a token here would result in a Connect 4 for player 1, it's a <code>1</code>, otherwise a <code>0</code></li>
  <li>For the second value, if placing a token would get player 2 a Connect 4, then it's a <code>1</code> otherwise a <code>0</code></li>
  <li>So if no one would win by placing a token here, it's two <code>0</code> values for these two inputs</li>
  </ul></li>
  <li>Two final values, the first indicating whether it's player 1's turn or not, then second indicating if it's player 2's turn or not</li>
  </ol>
  
  <p>What I see when my bot is playing is that it makes what I assume are somewhat clever moves, like it's preparing for both horizontal and diagonal moves.</p>
  
  <p>But when the other bot will get a Connect 4 on the next move, my bot fails to place a token there. The best bot in the comp seems capable of setting itself up to get three in a row with a free space on both sides, so that it will definitely get a Connect 4. Again, my bot does not seem to be able to see this coming.</p>
  
  <p>What I think the issue is, other than potentially my learning rates, is that I've represented the board in a bad way. Is there a better way to represent it to the Network so that it can more accurately estimate the value of a board state, and so that it doesn't fail to identify an immediate Connect 4 threat?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-02T20:01:48.690Z
Comments.Text: When you say "back propagate" you mean via the eligibility trace or lambda return, in order to set the TD target? Or do you literally mean that you back propagate reward signals to the NN for the end position? Are you also using some kind of forward/planning search with alpha/beta pruning?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-02T22:04:44.590Z
Comments.Text: Hi @neil I'm back propagating after every move, NN plays both sides. I adjust the weights and eligibility traces for the current state using the previous states values according to https://web.archive.org/web/20100625030049/http://www.cse.unr.edu/robotics/bekris/cs482_f09/sites/cse.unr.edu.robotics.bekris.cs482_f09/files/backgammon.pdf At the terminal state I change 'y1' in the above equations to be either `0` for a draw or `+-1` for win/lose. I use a 2-ply search to find the best move to make based upon the average value of all the opponents possible response move.
Comments.UserDisplayName: NeomerArcana
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-02T22:05:12.050Z
Comments.Text: Oh and I reset eligibility traces after every game.
Comments.UserDisplayName: NeomerArcana
Tags: <neural-networks><game-ai><combinatorial-games>
Title: Is this a good way to represent Connect 4 to a Neural Network?
UserDisplayName: NeomerArcana
ThreadText: # Is this a good way to represent Connect 4 to a Neural Network?
  
  (Asked by: username_0 on 2017-12-01)
  
  I'm attempting to make a bot for the Connect 4 competition on <http://riddles.io>
  
  My bot isn't horrible, like it's getting up the ladder, but it cannot compete with the winning bots.
  
  I'm using a Neural Network which is fully connected with one hidden layer. Internally it uses the sigmoid function as the activator in each Neuron. I've trained it over 500,000 games with TD-Lambda back propagation. The alpha and beta values (i.e. learning rates) are set to `0.1` each, and the lambda for the eligibility trace is set to `0.7`. There are 2 outputs nodes, one to give the value of this position for Player 1, the other to give the value of the position for Player 2. Upon a win, these are back-propagated with `-1` for a loss and `1` for a win, on a draw they are both back-propagated with a `0`.
  
  There is a bias input and weight for every neuron as well.
  
  All of the weights are initialised to a random value between `+-4*sqrt(6/totalNumberWeights)`.
  
  The board state is represented to the network as:
  
  1. For each space in the board, 2 values:
     * For the first value if player 1 occupies this space it's a `1`, otherwise it's a `0`
     * For the second value if player 2 occupies this space it's a `1`, otherwise it's a `0`
     * If both are `0` it would mean it's a free spot
  2. For each space on the board another 2 values:
     * For the first value, if placing a token here would result in a Connect 4 for player 1, it's a `1`, otherwise a `0`
     * For the second value, if placing a token would get player 2 a Connect 4, then it's a `1` otherwise a `0`
     * So if no one would win by placing a token here, it's two `0` values for these two inputs
  3. Two final values, the first indicating whether it's player 1's turn or not, then second indicating if it's player 2's turn or not
  
  What I see when my bot is playing is that it makes what I assume are somewhat clever moves, like it's preparing for both horizontal and diagonal moves.
  
  But when the other bot will get a Connect 4 on the next move, my bot fails to place a token there. The best bot in the comp seems capable of setting itself up to get three in a row with a free space on both sides, so that it will definitely get a Connect 4. Again, my bot does not seem to be able to see this coming.
  
  What I think the issue is, other than potentially my learning rates, is that I've represented the board in a bad way. Is there a better way to represent it to the Network so that it can more accurately estimate the value of a board state, and so that it doesn't fail to identify an immediate Connect 4 threat?
  
  **Tags:** neural-networks, game-ai, combinatorial-games
  
  > **Comments:**
  > **username_1**: When you say "back propagate" you mean via the eligibility trace or lambda return, in order to set the TD target? Or do you literally mean that you back propagate reward signals to the NN for the end position? Are you also using some kind of forward/planning search with alpha/beta pruning?  
  > **username_0**: Hi @username_1 I'm back propagating after every move, NN plays both sides. I adjust the weights and eligibility traces for the current state using the previous states values according to https://web.archive.org/web/20100625030049/http://www.cse.unr.edu/robotics/bekris/cs482_f09/sites/cse.unr.edu.robotics.bekris.cs482_f09/files/backgammon.pdf At the terminal state I change 'y1' in the above equations to be either `0` for a draw or `+-1` for win/lose. I use a 2-ply search to find the best move to make based upon the average value of all the opponents possible response move.  
  > **username_0**: Oh and I reset eligibility traces after every game.  
  
  ---
  
  ## 0 answers
  
  