Body: <p>I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to try to make a replica of that, then create a neural network and AI to play against me (Or at least something where I can enter the current board scenario, and it will output which row is the best move). This may be an ambitious first project, but I'm willing to put in the work and research to create something I'm proud of. I created the game using p5.js, and though it may be simple, I'm really happy with how it turned out, as it's one of my first more interesting and unique projects in computer science. Now I don't know a ton about ML, so bear with me. I would like the use pytorch, but I'm open to tensorflow/keras as well. </p>
  
  <p>Here are a few of my questions:</p>
  
  <ol>
  <li>What output do I need to train? My game currently doesn't have a win condition. Would an array or matrix filled with a 0 where there isn't a chip, a 1 where a red chip is, and a 2 for a yellow? Ie</li>
  </ol>
  
  <pre><code>[0,0,0,0
   1,0,0,0
   1,0,0,0
   1,0,2,0
   1,2,2,2]
  </code></pre>
  
  <p>and enter a 1 somewhere to signify this as a win for player 1? Could an AI recognize this 4 in a row pattern as what needs to be done to win? </p>
  
  <ol start="2">
  <li><p>What is the best way to simulate a lot of games to get my training data? I'm imagining using an RNG to drop chips randomly, export the data output to a file and then enter whether it was a win for p1, p2, or a tie?</p></li>
  <li><p>Any other general words of wisdom or links to read?</p></li>
  </ol>
  
  <p>Thanks so much for reading this and any help you can offer?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-02T02:53:54.530Z
Comments.Text: On your first question, numerically for an AI,the difference between a 1 and a 2 could be difficult for it to decipher. Instead, I would suggest using a CNN and adding 2 different channels to the input, one for player 1 and another for player 2, then set the values in these channels to be a 1 for a chip or a 0 otherwise. This will form a sort of image of the board where, if overlayed, you might get like a blue pixel for player 1 and a red pixel for player 2, and a CNN could perform quite well on that.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-02T03:21:56.440Z
Comments.Text: @Recessive I will definitely look into that. Out of curiosity, why is it better to do this as an image and then turn it into data rather than just feed it data like a matrix? Would I input whole games or one move at a time? Are you saying to do one image for player 1 chips and 1 image for player 2 chips, overlay them and keep the others empty? Sorry I'm new at this but I appreciate the help!
Comments.UserDisplayName: joelanbanks3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-02T03:30:42.940Z
Comments.Text: While CNN's are designed for images, they work exceptionally well on any 2 dimensional data, as you can imagine any matrix with 2 dimensions as a really concise image. In this case, the data is still being fed in as a matrix, just into a CNN as CNN's handle 2d data very well. The data isn't changed in any way, it's only being looked at from a different perspective. Also, no, feeding whole games would be redundant, so just the current board state will do for determining a move. As for you last question - sort of. Search up "Channels in CNN's" to get a better understanding of this.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-02T14:57:49.850Z
Comments.Text: @Recessive Ok so you're saying use images of the board 1 move at a time? I'm looking at CNNs and yes, they seem like the best thing to use. So if I understand correctly, normally with CNNs you have to assign each image an output, like with the fashion database you assign each one a class, what would the outputs/classes be for each move?
Comments.UserDisplayName: joelanbanks3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-03T03:39:20.647Z
Comments.Text: Very good question. This is where it gets tough. If you wanted to do it best, you would use a https://en.wikipedia.org/wiki/Monte_Carlo_tree_search (a sort of modified minimax). However this might prove difficult. For the sake of simplicity, I would recommend just representing all board positions as a big output vector (7x6, so 42 output nodes) with each representing the next move. Using softmax, only one will be chosen.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-03T03:42:58.370Z
Comments.Text: Of course, this poses the problem of how you are going to train it. If you wanted to do it simply, you could cheat and use an already existing AI to determine the next best move, and using supervised learning, tell your AI what moves it should be making. If you wanted to be more legitimate with the training however, reinforcement learning is your best bet. Unfortunately, I am not very experienced with reinforcement learning, so can't help out heaps there, but I know enough to know it would be applicable in this circumstance.
Comments.UserDisplayName: Recessive
Tags: <reinforcement-learning><training>
Title: What form of output would be needed to train a model on a connect 4 AI?
UserDisplayName: joelanbanks3
ThreadText: # What form of output would be needed to train a model on a connect 4 AI?
  
  (Asked by: username_0 on 2019-12-02)
  
  I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to try to make a replica of that, then create a neural network and AI to play against me (Or at least something where I can enter the current board scenario, and it will output which row is the best move). This may be an ambitious first project, but I'm willing to put in the work and research to create something I'm proud of. I created the game using p5.js, and though it may be simple, I'm really happy with how it turned out, as it's one of my first more interesting and unique projects in computer science. Now I don't know a ton about ML, so bear with me. I would like the use pytorch, but I'm open to tensorflow/keras as well. 
  Here are a few of my questions:
  
  What output do I need to train? My game currently doesn't have a win condition. Would an array or matrix filled with a 0 where there isn't a chip, a 1 where a red chip is, and a 2 for a yellow? Ie
  
  [0,0,0,0
   1,0,0,0
   1,0,0,0
   1,0,2,0
   1,2,2,2]
  
  and enter a 1 somewhere to signify this as a win for player 1? Could an AI recognize this 4 in a row pattern as what needs to be done to win? 
  
  What is the best way to simulate a lot of games to get my training data? I'm imagining using an RNG to drop chips randomly, export the data output to a file and then enter whether it was a win for p1, p2, or a tie?
  Any other general words of wisdom or links to read?
  
  Thanks so much for reading this and any help you can offer?
  
  **Tags:** reinforcement-learning, training
  > **Comments:**
  > **username_1**: On your first question, numerically for an AI,the difference between a 1 and a 2 could be difficult for it to decipher. Instead, I would suggest using a CNN and adding 2 different channels to the input, one for player 1 and another for player 2, then set the values in these channels to be a 1 for a chip or a 0 otherwise. This will form a sort of image of the board where, if overlayed, you might get like a blue pixel for player 1 and a red pixel for player 2, and a CNN could perform quite well on that.  
  > **username_0**: @Recessive I will definitely look into that. Out of curiosity, why is it better to do this as an image and then turn it into data rather than just feed it data like a matrix? Would I input whole games or one move at a time? Are you saying to do one image for player 1 chips and 1 image for player 2 chips, overlay them and keep the others empty? Sorry I'm new at this but I appreciate the help!  
  > **username_1**: While CNN's are designed for images, they work exceptionally well on any 2 dimensional data, as you can imagine any matrix with 2 dimensions as a really concise image. In this case, the data is still being fed in as a matrix, just into a CNN as CNN's handle 2d data very well. The data isn't changed in any way, it's only being looked at from a different perspective. Also, no, feeding whole games would be redundant, so just the current board state will do for determining a move. As for you last question - sort of. Search up "Channels in CNN's" to get a better understanding of this.  
  > **username_0**: @Recessive Ok so you're saying use images of the board 1 move at a time? I'm looking at CNNs and yes, they seem like the best thing to use. So if I understand correctly, normally with CNNs you have to assign each image an output, like with the fashion database you assign each one a class, what would the outputs/classes be for each move?  
  > **username_1**: Very good question. This is where it gets tough. If you wanted to do it best, you would use a https://en.wikipedia.org/wiki/Monte_Carlo_tree_search (a sort of modified minimax). However this might prove difficult. For the sake of simplicity, I would recommend just representing all board positions as a big output vector (7x6, so 42 output nodes) with each representing the next move. Using softmax, only one will be chosen.  
  > **username_1**: Of course, this poses the problem of how you are going to train it. If you wanted to do it simply, you could cheat and use an already existing AI to determine the next best move, and using supervised learning, tell your AI what moves it should be making. If you wanted to be more legitimate with the training however, reinforcement learning is your best bet. Unfortunately, I am not very experienced with reinforcement learning, so can't help out heaps there, but I know enough to know it would be applicable in this circumstance.  
  
  ---
  
  ## 0 answers
  
  