Answers.Body: <p>The following are extremely simple ways of tackling this problem.</p>
  <h2>A very simple way</h2>
  <p>It can simply be<br/>
  <code>strength of AI</code>=<code>(# of games won)</code>/<code>(total # of games)</code>.</p>
  <br/>
  <h2>In case data for each move is available</h2>
  <p>Something like<br/>
  <code>score per game</code>=<code># of correct decisions</code>/<code>total number of decisions</code>.<br/>
  Then<br/>
  <code>strength of AI</code>=<code>sum(score per game)</code>/<code>total # of games</code>.</p>
  <br/>
  <h2>If each move/decision has a score associated with it</h2>
  <p>then you do<br/><code>score per move</code>=<code>scored points by taking a decision</code>/<code>maximum possible score</code>.<br/>
  then<br/>
  <code>score per game</code>=<code>sum(score per move)</code>/<code>total # of moves</code><br/>
  and finally,<br/>
  <code>strength of AI</code>=<code>sum(score per game)</code>/<code>total # of games</code>.</p>
  <br/>
  <h2>How to choose optimal number of games to play?</h2>
  <p>It depends on your requirements. If you want to report your AI's strength in percentage of the games it played correct to 1 decimal place (for example, this AI won 95.1%) then 10000 is an optimal number of games your AI needs to play. Suppose your AI won 9508 games out of 10000 then you will have 95.08% strength of AI. To be able to correctly round it to 1 decimal place you need to have an additional decimal place so that you can quote the strength of you AI with reasonable confidence, in this case 95.1%.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-25T21:23:54.197Z
Answers.Text: Can you explain how to select optimal number of total games  and how many people to engage for the experiment?
Answers.UserDisplayName: tamirok
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T05:55:18.387Z
Answers.Text: I have tried to explain the optimal number of total games in the edit of the answer above.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T05:55:38.517Z
Answers.Text: I am not sure how you can determine the optimal number of people to choose.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T17:36:14.190Z
Answers.Text: @DuttaA That's true, most probably. It very well looks like it as well.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T17:38:41.383Z
Answers.Text: This experiment can be repeated against multiple people who are aware of how to play the game. Then average strength can be reported as the final one.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T17:39:20.717Z
Answers.Text: The experiment can also be repeated with only one top tier human player and an average can be reported
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-25T09:13:30.940Z
Answers.LastActivityDate: 2018-03-26T05:54:52.317Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Mohsin
Body: <p>I have implemented multiple MCTS based AI players for the Love Letter game (<a href="https://en.wikipedia.org/wiki/Love_Letter_(card_game)" rel="nofollow noreferrer">rules</a>). It is a 2-4 players zero sum card game where players make alternating moves. I am struggling with how to properly conduct experiments for estimating AI player strength against human players: </p>
  
  <ol>
  <li>In 2 player game where one of the players is AI bot</li>
  <li>In 4 player game where one (or multiple) of players is AI bot</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-25T08:36:23.563Z
Comments.Text: Do you have data (preferably lots) regarding human play choices? Do you have some humans available to help testing?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-25T21:19:37.280Z
Comments.Text: @NeilSlater I don't have human play data. However, I have some humans availlable to help me in testing. Is it good approach to take 15 players (including 1 AI bot)  with initial Elo rating and let them play against each other?
Comments.UserDisplayName: tamirok
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-26T18:50:41.867Z
Comments.Text: If you are only concerned with win/loss outcomes, you can use the ELO system.
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Tags: <game-ai><monte-carlo-tree-search>
Title: How to estimate the AI player's strength in multiplayer game?
UserDisplayName: tamirok
ThreadText: # How to estimate the AI player's strength in multiplayer game?
  
  (Asked by: username_0 on 2018-03-24)
  
  I have implemented multiple MCTS based AI players for the Love Letter game ([rules](https://en.wikipedia.org/wiki/Love_Letter_(card_game))). It is a 2-4 players zero sum card game where players make alternating moves. I am struggling with how to properly conduct experiments for estimating AI player strength against human players:
  
  1. In 2 player game where one of the players is AI bot
  2. In 4 player game where one (or multiple) of players is AI bot
  
  **Tags:** game-ai, monte-carlo-tree-search
  > **Comments:**
  > **username_1**: Do you have data (preferably lots) regarding human play choices? Do you have some humans available to help testing?  
  > **username_0**: @NeilSlater I don't have human play data. However, I have some humans availlable to help me in testing. Is it good approach to take 15 players (including 1 AI bot)  with initial Elo rating and let them play against each other?  
  > **username_2**: If you are only concerned with win/loss outcomes, you can use the ELO system.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-03-25. Score: 2
  
  The following are extremely simple ways of tackling this problem.
  
  ## A very simple way
  
  It can simply be  
  `strength of AI`=`(# of games won)`/`(total # of games)`.
  
    
  
  ## In case data for each move is available
  
  Something like  
  `score per game`=`# of correct decisions`/`total number of decisions`.  
  Then  
  `strength of AI`=`sum(score per game)`/`total # of games`.
  
    
  
  ## If each move/decision has a score associated with it
  
  then you do  
  `score per move`=`scored points by taking a decision`/`maximum possible score`.  
  then  
  `score per game`=`sum(score per move)`/`total # of moves`  
  and finally,  
  `strength of AI`=`sum(score per game)`/`total # of games`.
  
    
  
  ## How to choose optimal number of games to play?
  
  It depends on your requirements. If you want to report your AI's strength in percentage of the games it played correct to 1 decimal place (for example, this AI won 95.1%) then 10000 is an optimal number of games your AI needs to play. Suppose your AI won 9508 games out of 10000 then you will have 95.08% strength of AI. To be able to correctly round it to 1 decimal place you need to have an additional decimal place so that you can quote the strength of you AI with reasonable confidence, in this case 95.1%.
  
  > **Comments:**
  > **username_0**: Can you explain how to select optimal number of total games  and how many people to engage for the experiment?  
  > **username_3**: I have tried to explain the optimal number of total games in the edit of the answer above.  
  > **username_3**: I am not sure how you can determine the optimal number of people to choose.  
  > **username_3**: @DuttaA That's true, most probably. It very well looks like it as well.  
  > **username_3**: This experiment can be repeated against multiple people who are aware of how to play the game. Then average strength can be reported as the final one.  
  > **username_3**: The experiment can also be repeated with only one top tier human player and an average can be reported  
  
  ---
  
  