<p>The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay. </p>

CC BY-SA 4.0
2019-12-05T22:14:33.797Z
Thanks, how do you mean the evaluation of the best next step? For example, if a simulated game of the neural network vs itself [proceeds like this](https://i.imgur.com/jzBhWGv.png). The last move of A (in step 7) should obviously be scored 1 because it wins the game, and the preceding move of B (in step 6) should score 0 because it allowed the opponent to win. But then how further? The preceding step directly before that, move 5 by player A, was actually a very bad move, as player B could have won after that. And in fact player A started the game very badly. So how do I further assign scores?
RocketNuts
CC BY-SA 4.0
2019-12-07T09:25:36.390Z
You should also score it 1(maybe with decay, so 0,99), because it lead to a win. This will make it take this wrong decission more in the future though. Eventually B will learn how to defeat this move and at that point, you wil start scoring step 5 with 0 because it lost that game.
Lustwelpintje
CC BY-SA 4.0
2019-12-07T09:29:01.813Z
This means that the early steps won't be properly trained at the start of the training sessions, but given enough training games, eventually he will learn how to correctly do the early steps
Lustwelpintje
CC BY-SA 4.0
2019-12-08T19:07:04.780Z
OK I see, so this means I should rate A's moves 7, 5, 3, 1 with a score of 1, 0.99, 0.98, 0.97 respectively (or 0.99², 0.99³, etc) and B's moves 6, 4, 2 with a score of 0, 0.01, 0.02 respectively (or 1-0.99ⁿ). But one more thing: in this case A and B are actually the same neural network. So do I take the average of all the weight adjustments, i.e. learning rate η times 0.99ⁿ for the moves on the winning side (with n = number of remaining moves until the end of the game for that side) and η(1-0.99ⁿ) for the moves on the losing side. And then backpropagate the resulting average score adjustment?
RocketNuts
CC BY-SA 4.0
2019-12-09T10:29:39.023Z
you just train every move once right? I don't really know why you want to average the weight adjustments?
Lustwelpintje
CC BY-SA 4.0
2019-12-09T10:37:56.257Z
Ah, OK no I thought you meant some kind of (mini-)batch training and then apply weight adjustments once per game. Although now that I think about it, I guess it actually boils down to the same. So yes I can just adjust the weights for every individual move. Thanks again, much appreciated.
RocketNuts
CC BY-SA 4.0
2024-05-17T20:13:13.990Z
Was this project completed?  I am trying to understand to what extent tree search is required for play, as opposed to something much easier like you describe: loop through every next move, pick best.
Manu
CC BY-SA 4.0
2019-12-03T14:52:32.130Z
2019-12-03T14:52:32.130Z
Lustwelpintje
<p>I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them <strong>evaluate a position</strong>. </p>

<p>The input is any board situation. The output is a score or estimate for the probability to win, or how favorable a given position is. Where 1 = guaranteed to win and 0 = guaranteed to lose.</p>

<p>In any given turn I can then loop over all possible moves for the current player, evaluate the resulting game situations, and pick the one with the highest score. </p>

<p>Hopefully, by letting this neural network play a trillion games vs itself, it can develop a sensible scoring function resulting in strong play. </p>

<p>Question: <strong>How do I train such a network</strong> ?</p>

<p>In every game, I can keep evaluating and making moves back and forth until one of the AI players wins. In that case the last game situation (right before the winning move) for the winning player should have a target value of 1, and the opposite situation (for the losing player) has a target value of 0. </p>

<p>Note that I don't intend to make the evaluation network two-sided. I encode the game situation as always "my own" pieces vs "the opponent", and then evaluate a score from my own (i.e. the current player's) side or perspective. Then after I pick a move, I flip sides so to speak, so the opponent pieces now become my own and vice versa, and then evaluate scores again (now from the other side's perspective) for the next counter move.</p>

<p>So the input to such a network does explicitly encode black and white pieces, or naughts and crosses (in case of tic-tac-toe) but just my pieces vs their pieces. And then evaluates how favorable the given game situation is for me, always assuming it's my turn.</p>

<p>I can obviously assign a desired score or truth value for the last move in the game (1 for win, 0 for loss) but how do I backpropagate that towards earlier situations in a played game?</p>

<p>Should I somehow distribute the 1 or 0 result back a few steps, with a decaying adjustment factor or learning rate? In a game with 40 turns, it might make sense to consider the last few situations as good or bad (being close to winning or losing) but I guess that shouldn't reflect all the way back to the first few moves in the game. </p>

<p>Or am I completely mistaken with this approach and is this not how it's supposed to be done?</p>

CC BY-SA 4.0
2019-12-02T14:57:44.653Z
This idea is on the right track! you have pretty much defined alphaZero, where they train a NN to evaluate each move, and then use MC-search for the actual play throughs!
mshlis
CC BY-SA 4.0
2019-12-02T20:45:24.647Z
Ok thanks, and do you know if I can find some explanation or further documentation on the training process?
RocketNuts
<neural-networks><reinforcement-learning><game-ai><software-evaluation><alphazero>
Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?
RocketNuts
# Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?

(Asked by: username_0 on 2019-12-02)

I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them **evaluate a position**.

The input is any board situation. The output is a score or estimate for the probability to win, or how favorable a given position is. Where 1 = guaranteed to win and 0 = guaranteed to lose.

In any given turn I can then loop over all possible moves for the current player, evaluate the resulting game situations, and pick the one with the highest score.

Hopefully, by letting this neural network play a trillion games vs itself, it can develop a sensible scoring function resulting in strong play.

Question: **How do I train such a network** ?

In every game, I can keep evaluating and making moves back and forth until one of the AI players wins. In that case the last game situation (right before the winning move) for the winning player should have a target value of 1, and the opposite situation (for the losing player) has a target value of 0.

Note that I don't intend to make the evaluation network two-sided. I encode the game situation as always "my own" pieces vs "the opponent", and then evaluate a score from my own (i.e. the current player's) side or perspective. Then after I pick a move, I flip sides so to speak, so the opponent pieces now become my own and vice versa, and then evaluate scores again (now from the other side's perspective) for the next counter move.

So the input to such a network does explicitly encode black and white pieces, or naughts and crosses (in case of tic-tac-toe) but just my pieces vs their pieces. And then evaluates how favorable the given game situation is for me, always assuming it's my turn.

I can obviously assign a desired score or truth value for the last move in the game (1 for win, 0 for loss) but how do I backpropagate that towards earlier situations in a played game?

Should I somehow distribute the 1 or 0 result back a few steps, with a decaying adjustment factor or learning rate? In a game with 40 turns, it might make sense to consider the last few situations as good or bad (being close to winning or losing) but I guess that shouldn't reflect all the way back to the first few moves in the game.

Or am I completely mistaken with this approach and is this not how it's supposed to be done?

**Tags:** neural-networks, reinforcement-learning, game-ai, software-evaluation, alphazero

> **Comments:**
> **username_1**: This idea is on the right track! you have pretty much defined alphaZero, where they train a NN to evaluate each move, and then use MC-search for the actual play throughs!  
> **username_0**: Ok thanks, and do you know if I can find some explanation or further documentation on the training process?  

---

## 1 answers

### Answer by username_2 on 2019-12-03. Score: 1 | Accepted answer

The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay.

> **Comments:**
> **username_0**: Thanks, how do you mean the evaluation of the best next step? For example, if a simulated game of the neural network vs itself [proceeds like this](https://i.imgur.com/jzBhWGv.png). The last move of A (in step 7) should obviously be scored 1 because it wins the game, and the preceding move of B (in step 6) should score 0 because it allowed the opponent to win. But then how further? The preceding step directly before that, move 5 by player A, was actually a very bad move, as player B could have won after that. And in fact player A started the game very badly. So how do I further assign scores?  
> **username_2**: You should also score it 1(maybe with decay, so 0,99), because it lead to a win. This will make it take this wrong decission more in the future though. Eventually B will learn how to defeat this move and at that point, you wil start scoring step 5 with 0 because it lost that game.  
> **username_2**: This means that the early steps won't be properly trained at the start of the training sessions, but given enough training games, eventually he will learn how to correctly do the early steps  
> **username_0**: OK I see, so this means I should rate A's moves 7, 5, 3, 1 with a score of 1, 0.99, 0.98, 0.97 respectively (or 0.99², 0.99³, etc) and B's moves 6, 4, 2 with a score of 0, 0.01, 0.02 respectively (or 1-0.99ⁿ). But one more thing: in this case A and B are actually the same neural network. So do I take the average of all the weight adjustments, i.e. learning rate η times 0.99ⁿ for the moves on the winning side (with n = number of remaining moves until the end of the game for that side) and η(1-0.99ⁿ) for the moves on the losing side. And then backpropagate the resulting average score adjustment?  
> **username_2**: you just train every move once right? I don't really know why you want to average the weight adjustments?  
> **username_0**: Ah, OK no I thought you meant some kind of (mini-)batch training and then apply weight adjustments once per game. Although now that I think about it, I guess it actually boils down to the same. So yes I can just adjust the weights for every individual move. Thanks again, much appreciated.  
> **username_3**: Was this project completed?  I am trying to understand to what extent tree search is required for play, as opposed to something much easier like you describe: loop through every next move, pick best.  

---

