Body: <p>I have implemented an AI agent to play checkers based on the design written in the first chapter of <a href="https://rads.stackoverflow.com/amzn/click/com/0070428077" rel="nofollow noreferrer" rel="nofollow noreferrer">
  Machine Learning, Tom Mitchell, McGraw Hill, 1997.</a></p>
  <p>We train the agent by letting it plays against its self.</p>
  <p>I wrote the prediction to get how good a board is for white, so when the white plays he must choose the next board with the maximum value, and when black plays it must choose the next board with the minimum value.</p>
  <p>Also, I let the agent explore other states by making it choose a random board among the valid next boards, I let that probability to be equal to <span class="math-container">$0.1$</span>.</p>
  <p>The final boards will have training values:</p>
  <blockquote>
  <p>100 if this final board is <code>win</code> for white.</p>
  </blockquote>
  <blockquote>
  <p>-100 if this final board is <code>lose</code> for white.</p>
  </blockquote>
  <blockquote>
  <p>0 if this final board is <code>draw</code>.</p>
  </blockquote>
  <p>The intermediate boards will have a training value equal to the prediction of the next board where it is white turn.</p>
  <p>The model is based on a linear combination of some features (see the book for full description).</p>
  <p>I start by initializing the parameters of the model to random values.</p>
  <p>When I train the agent, he lost against himself always or draw in a stupid way, but the error converges to zero.</p>
  <p>I was thinking that maybe we should let the learning rate smaller (like 1e-5), and when I do that the agent learns in a better way.</p>
  <p>I think that this happened because of the credit assignment problem, because a good move may appear in a loose game, therefore, considered a loose move, so white will never choose it when he plays, but when we let the learning rate to be very small that existence of a good move in a losing game will change its value by a very little amount, and that good move should appear more in win games so its value converges to the right value.</p>
  <p>Is my reasoning correct? and if not so what is happened?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T18:04:14.023Z
Comments.Text: Please, put your main **specific** question in the title. "AI agent playing checkers" is not a question and it's very general. If you have more than one specific question, split this post into multiple ones (one for each of the questions that you have).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T18:29:47.143Z
Comments.Text: @nbro sorry for that, I edited the question, I hope it is more clear now.
Comments.UserDisplayName: Kais Hasan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-02T21:12:42.287Z
Comments.Text: There are a couple of things that might be off here. Number one is, as you mentioned the learning rate. Number two is the exploration policy (i.e. taking a random action only 10% of times), you might want to try and have a higher chance of random move for a while and then progressively lower it. Number 3 is that the agent is playing against itself, but when he's black his aim is to lose. Is the agent fed the information of who he is (black or white) as part of the observation? This would be essential
Comments.UserDisplayName: Federico Malerba
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-03T09:45:32.383Z
Comments.Text: @FedericoMalerba I tried the second point you mentioned but that does not solve the problem.
  I do not understand the third point, the prediction based on the evaluation of the board for white, so black need to minimize the board goodness for white, so he choose the minimum.
  For the training, I only take the boards when it's white's turn so no need for the colour.
  I take this from the book I mentioned at the top of the question.
Comments.UserDisplayName: Kais Hasan
Tags: <machine-learning><reinforcement-learning><intelligent-agent><checkers><credit-assignment-problem>
Title: learning rate and credit assignment problem in checkers
UserDisplayName: Kais Hasan
ThreadText: # learning rate and credit assignment problem in checkers
  
  (Asked by: username_0 on 2020-12-31)
  
  I have implemented an AI agent to play checkers based on the design written in the first chapter of [Machine Learning, Tom Mitchell, McGraw Hill, 1997.](https://rads.stackoverflow.com/amzn/click/com/0070428077)
  
  We train the agent by letting it plays against its self.
  
  I wrote the prediction to get how good a board is for white, so when the white plays he must choose the next board with the maximum value, and when black plays it must choose the next board with the minimum value.
  
  Also, I let the agent explore other states by making it choose a random board among the valid next boards, I let that probability to be equal to $0.1$.
  
  The final boards will have training values:
  
  > 100 if this final board is `win` for white.
  
  > -100 if this final board is `lose` for white.
  
  > 0 if this final board is `draw`.
  
  The intermediate boards will have a training value equal to the prediction of the next board where it is white turn.
  
  The model is based on a linear combination of some features (see the book for full description).
  
  I start by initializing the parameters of the model to random values.
  
  When I train the agent, he lost against himself always or draw in a stupid way, but the error converges to zero.
  
  I was thinking that maybe we should let the learning rate smaller (like 1e-5), and when I do that the agent learns in a better way.
  
  I think that this happened because of the credit assignment problem, because a good move may appear in a loose game, therefore, considered a loose move, so white will never choose it when he plays, but when we let the learning rate to be very small that existence of a good move in a losing game will change its value by a very little amount, and that good move should appear more in win games so its value converges to the right value.
  
  Is my reasoning correct? and if not so what is happened?
  
  **Tags:** machine-learning, reinforcement-learning, intelligent-agent, checkers, credit-assignment-problem
  > **Comments:**
  > **username_1**: Please, put your main **specific** question in the title. "AI agent playing checkers" is not a question and it's very general. If you have more than one specific question, split this post into multiple ones (one for each of the questions that you have).  
  > **username_0**: @nbro sorry for that, I edited the question, I hope it is more clear now.  
  > **username_2**: There are a couple of things that might be off here. Number one is, as you mentioned the learning rate. Number two is the exploration policy (i.e. taking a random action only 10% of times), you might want to try and have a higher chance of random move for a while and then progressively lower it. Number 3 is that the agent is playing against itself, but when he's black his aim is to lose. Is the agent fed the information of who he is (black or white) as part of the observation? This would be essential  
  > **username_0**: @FedericoMalerba I tried the second point you mentioned but that does not solve the problem. I do not understand the third point, the prediction based on the evaluation of the board for white, so black need to minimize the board goodness for white, so he choose the minimum. For the training, I only take the boards when it's white's turn so no need for the colour. I take this from the book I mentioned at the top of the question.  
  
  ---
  
  ## 0 answers
  
  