Answers.Body: <p>You probably want to have a look at <strong>reinforcement learning</strong>. I will not be able to cover this complex topic in one post, but I will give you some key ideas (and yes, this is a simplification and experts of the field might have a valid point that there are exceptions to what I am writing):</p>
  <ol>
  <li>You are already half-way there. Reinforcement learning aims to predict the gain of each possible action.</li>
  <li>What - from my understanding of your description - might be missing is the fact, that reinforcement learning how to estimate the gain of each action. This is done by evaluating the outcome of each game. You can image an iterative process:<br />
  Actions / States that lead to winning the game will be rates positive, those that lead to losing the game are negative. If this is learned, then actions that lead to a positive state will be rated positive and others will be rated negative.<br />
  By doing so, the system can learn the long-term effect of actions even though they have now direct gain. In your case, it might learn that having these special cards is a better state than not having them (and also when it is a good action to play them)</li>
  <li>Reinforcement learning can deal with uncertainties (e.g. the unknown cards of the other players, what kind of action they might take). A well designed reinforcement learning system optimizes for the best expectation.</li>
  <li>For the training, the system has to play against other computer players. The do not necessarily have to use the same strategy.</li>
  </ol>
  <p>For further insides and practical applications I would suggest to read some tutorials, introductions or books about the topic. As I said, the topic is complex.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T15:36:09.670Z
Answers.Text: Hi, thank you for your reply.
  I managed to make it work without those special cards. It correctly analyzes the state and does the correct moves.
  I want to ask about a certain special example.
  There is a very powerful card, but to play it, you need to play specific cards before.
  I am able to reflect the presence of those requirements in the game state. But how do I put the presence of that powerful card. Is it as simple as 0 =don't have it, 1=it is in your hand, 2=you played it? It must find a connection between requirements present in the state and that powerful card having a status 2?
Answers.UserDisplayName: Alex Nikitin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T15:39:51.290Z
Answers.Text: Also how to make a correct prediction for the multiple steps?
  For example I can play a cheap card now and get 2% extra win rate. Or I can play the "gain income", get more money, then play a powerful card and have 20% extra win rate? Does it mean that I should analyze the state not just 1 turn ahead, but sometimes even 2 turns ahead? The issue is, that opponent will also make not just 1, but 2 turns. And because I don't know his turns, going for those 20% instead of 2% in fact can me get into a worse situation, than I initially expected.
Answers.UserDisplayName: Alex Nikitin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T15:43:01.907Z
Answers.Text: Or another example. Making step 1 will reduce my win probability. But making step 1+step 2 overall will increase the win probability. If I analyse 1 step ahead, the neural network will say that step 1 will make the situation worse and I won't play it. But in fact as a human it would be a smart solution to play step 1, make the situation worse, in order to then, with step 2 to make it even better.
Answers.UserDisplayName: Alex Nikitin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-17T09:18:02.780Z
Answers.Text: This is exactly, what reinforcement learning does. It learns long-term effects and uncertainties of actions. You do not specify by yourself what the gain of getting / playing special cards is, but mainly define the rating at the end of the game.
Answers.UserDisplayName: Broele
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-17T09:20:50.837Z
Answers.Text: So for reinforcement learning, you need a good representation of the current state (i.e. everything you know about the current situation: what cards you have, what is played on the table, ...). Unknow information is left out (e.g. which cards other players have on their hand). In addition, you need a representation of all possible actions. What a good or bad representaion is depends on the game and the reinforcement algorithms that you use.
Answers.UserDisplayName: Broele
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-17T09:23:32.417Z
Answers.Text: Maybe you can first read some books or tutorials and try some simpler examples to get some experience and intuition of how reinforcement learning works?
Answers.UserDisplayName: Broele
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T10:06:58.110Z
Answers.LastActivityDate: 2023-03-11T10:06:58.110Z
Answers.UserDisplayName: Broele
Body: <p>I am a Software Engineer and I implemented online a pretty complex board game.
  Some key rules of the game.</p>
  <ol>
  <li>You have some cards in the hand and you can choose different turns.</li>
  <li>Build a card of certain type, which would increase your income or give victory points.</li>
  <li>You can choose to receive the resources equal to your income.</li>
  <li>You can choose to take more cards.</li>
  <li>Some cards that you build have pretty complex effects that can't be measured, some examples are: increase the number of cards you can take during a single turn, a card that lets you make a special turn if you have that card etc.</li>
  </ol>
  <p>What I already tried to do.
  I tried to create a state of the game that would include</p>
  <ul>
  <li>income of every player</li>
  <li>current number of resources of every player</li>
  <li>current victory points</li>
  <li>the state of the game that affects how long it might take till the end</li>
  <li>the number of current turn, since it is variable and depends on the play</li>
  </ul>
  <p>I would collect this data for every game, then mark which data belongs to a player who won, and who lost and teach the model.</p>
  <p>Then I took this model, calculated every possible turn and future state if a player made that turn and then I would evaluate which of those turn would show the highest chance of a win. And then I would consider that turn the most optimal.</p>
  <p>The problem with this approach is that it doesn't evaluate those special cards with special effects, because they simply don't fit into that model.</p>
  <p>Also the most complex part is that player may choose which cards he wants to take and which cards he wants to discard.
  How can I teach the engine to evaluate the value of that card? For example a certain card may be extremely good but there might be some limitations according to player current situation. I don't know how to put this data into the model.</p>
  <p>Also I ignore the fact that opposite player can play something, even though I can evaluate his current game situation, because I can see everything he played, or has, except his hand.</p>
  <p>In the end I simply made my computer play vs itself and collected data on which cards played on which turn show the best win rate for a player in the end.</p>
  <p>But I want to make and teach a model, maybe not by evaluating the board, because I can't fit those complex cards, but maybe some other solution.
  Give me please any advice at which approaches/literate to look.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-11T14:52:35.783Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Tags: <game-ai>
Title: The neural network for a board game with somewhat imperfect information
UserDisplayName: Alex Nikitin
ThreadText: # The neural network for a board game with somewhat imperfect information
  
  (Asked by: username_0 on 2023-03-11)
  
  I am a Software Engineer and I implemented online a pretty complex board game.
  Some key rules of the game.
  
  1. You have some cards in the hand and you can choose different turns.
  2. Build a card of certain type, which would increase your income or give victory points.
  3. You can choose to receive the resources equal to your income.
  4. You can choose to take more cards.
  5. Some cards that you build have pretty complex effects that can't be measured, some examples are: increase the number of cards you can take during a single turn, a card that lets you make a special turn if you have that card etc.
  
  What I already tried to do.
  I tried to create a state of the game that would include
  
  * income of every player
  * current number of resources of every player
  * current victory points
  * the state of the game that affects how long it might take till the end
  * the number of current turn, since it is variable and depends on the play
  
  I would collect this data for every game, then mark which data belongs to a player who won, and who lost and teach the model.
  
  Then I took this model, calculated every possible turn and future state if a player made that turn and then I would evaluate which of those turn would show the highest chance of a win. And then I would consider that turn the most optimal.
  
  The problem with this approach is that it doesn't evaluate those special cards with special effects, because they simply don't fit into that model.
  
  Also the most complex part is that player may choose which cards he wants to take and which cards he wants to discard.
  How can I teach the engine to evaluate the value of that card? For example a certain card may be extremely good but there might be some limitations according to player current situation. I don't know how to put this data into the model.
  
  Also I ignore the fact that opposite player can play something, even though I can evaluate his current game situation, because I can see everything he played, or has, except his hand.
  
  In the end I simply made my computer play vs itself and collected data on which cards played on which turn show the best win rate for a player in the end.
  
  But I want to make and teach a model, maybe not by evaluating the board, because I can't fit those complex cards, but maybe some other solution.
  Give me please any advice at which approaches/literate to look.
  
  **Tags:** game-ai
  
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-03-11. Score: 2
  
  You probably want to have a look at **reinforcement learning**. I will not be able to cover this complex topic in one post, but I will give you some key ideas (and yes, this is a simplification and experts of the field might have a valid point that there are exceptions to what I am writing):
  
  1. You are already half-way there. Reinforcement learning aims to predict the gain of each possible action.
  2. What - from my understanding of your description - might be missing is the fact, that reinforcement learning how to estimate the gain of each action. This is done by evaluating the outcome of each game. You can image an iterative process:  
     Actions / States that lead to winning the game will be rates positive, those that lead to losing the game are negative. If this is learned, then actions that lead to a positive state will be rated positive and others will be rated negative.  
     By doing so, the system can learn the long-term effect of actions even though they have now direct gain. In your case, it might learn that having these special cards is a better state than not having them (and also when it is a good action to play them)
  3. Reinforcement learning can deal with uncertainties (e.g. the unknown cards of the other players, what kind of action they might take). A well designed reinforcement learning system optimizes for the best expectation.
  4. For the training, the system has to play against other computer players. The do not necessarily have to use the same strategy.
  
  For further insides and practical applications I would suggest to read some tutorials, introductions or books about the topic. As I said, the topic is complex.
  
  > **Comments:**
  > **username_0**: Hi, thank you for your reply. I managed to make it work without those special cards. It correctly analyzes the state and does the correct moves. I want to ask about a certain special example. There is a very powerful card, but to play it, you need to play specific cards before. I am able to reflect the presence of those requirements in the game state. But how do I put the presence of that powerful card. Is it as simple as 0 =don't have it, 1=it is in your hand, 2=you played it? It must find a connection between requirements present in the state and that powerful card having a status 2?  
  > **username_0**: Also how to make a correct prediction for the multiple steps? For example I can play a cheap card now and get 2% extra win rate. Or I can play the "gain income", get more money, then play a powerful card and have 20% extra win rate? Does it mean that I should analyze the state not just 1 turn ahead, but sometimes even 2 turns ahead? The issue is, that opponent will also make not just 1, but 2 turns. And because I don't know his turns, going for those 20% instead of 2% in fact can me get into a worse situation, than I initially expected.  
  > **username_0**: Or another example. Making step 1 will reduce my win probability. But making step 1+step 2 overall will increase the win probability. If I analyse 1 step ahead, the neural network will say that step 1 will make the situation worse and I won't play it. But in fact as a human it would be a smart solution to play step 1, make the situation worse, in order to then, with step 2 to make it even better.  
  > **username_2**: This is exactly, what reinforcement learning does. It learns long-term effects and uncertainties of actions. You do not specify by yourself what the gain of getting / playing special cards is, but mainly define the rating at the end of the game.  
  > **username_2**: So for reinforcement learning, you need a good representation of the current state (i.e. everything you know about the current situation: what cards you have, what is played on the table, ...). Unknow information is left out (e.g. which cards other players have on their hand). In addition, you need a representation of all possible actions. What a good or bad representaion is depends on the game and the reinforcement algorithms that you use.  
  > **username_2**: Maybe you can first read some books or tutorials and try some simpler examples to get some experience and intuition of how reinforcement learning works?  
  
  ---
  
  