Answers.Body: <p>When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of comparing it to humans, you might want to compare the game results of all the games played.</p>
  <h1>Reward Comparison</h1>
  <p>Often Reinforcement Learning algorithms are compared by using the rewards (either direct, maximum or average in time/iteration). For example, in <a href="https://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html" rel="nofollow noreferrer">this page about RL</a> a comparison of two algorithms is shown:</p>
  <p><a href="https://i.sstatic.net/zDk9w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zDk9w.png" alt="Comparing two RL algorithms" /></a></p>
  <p>Or, when you know the optimal actions, you can plot the number of plays/iterations against the percentage of actions. See for example this <a href="http://incompleteideas.net/book/first/ebook/node22.html" rel="nofollow noreferrer">RL comparison on the 10-armed testbed problem</a>:</p>
  <p><img src="https://i.sstatic.net/buIpx.png" alt="Comparing two RL algorithms with optimal actions" /></p>
  <p><a href="https://arxiv.org/abs/1709.06560" rel="nofollow noreferrer">Henderson et al. 2017</a> have a whole section about the evaluation metrics of Reinforcement Learning algorithms. They also comment on the plotting of the average or maximum cumulative rewards, moreover they mention the <em>sample bootstrap</em> method to create confidence intervals for a better comparison. Lastly, they mention that the <em>significance</em> of the improvements of the algorithms should be calculated, using a statistical test, such the <a href="https://stattrek.com/statistics/dictionary.aspx?definition=two-sample%20t-test" rel="nofollow noreferrer">two sample t-Test</a>. Note that you should take into account the distributions of the datasets to choose the right statistical test. An interesting article related to this is <a href="https://openreview.net/pdf?id=ryx0N3IaIV" rel="nofollow noreferrer">A Hitchhiker's Guide to Statistical Comparison of Reinforcement Learning Algorithms</a> of <a href="https://openreview.net/pdf?id=ryx0N3IaIV" rel="nofollow noreferrer">Colas et al.</a>.</p>
  <h1>Comparison of plays against humans</h1>
  <p>To find out how well different algorithms play against humans you should do a large number of games and compare - what you consider - important parameters, for example: did the algorithm won, the time it took to win, number of points gained, etc. These values can then be compared statistically. Note that you have to think well about the setup of these experiments since you only should change the algorithms, the other parameters should stay equal. Therefore, you should - preferably - use a large number of subjects (of different ages, sexes, etc. to cover many types of people), and try to prevent any bias; think about the order, how many games are played, location, time etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-13T22:13:00.000Z
Answers.LastActivityDate: 2019-08-13T22:13:00.000Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: agold
Body: <p>I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, &quot;Version X performed better than version Y, as we can see that Z is much better/higher/lower.&quot;</p>
  <p>I understand that we can use convergence for some RL algorithms, but, if the RL is playing against a human in the game, how am I able to evaluate its performance properly?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-31T21:35:14.760Z
Comments.Text: [Here](https://ai.stackexchange.com/q/20903/2444) you have a related question.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><rewards><performance><return><testing>
Title: How to evaluate an RL algorithm when used in a game?
UserDisplayName: mason7663
ThreadText: # How to evaluate an RL algorithm when used in a game?
  
  (Asked by: username_0 on 2019-08-10)
  
  I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as we can see that Z is much better/higher/lower."
  
  I understand that we can use convergence for some RL algorithms, but, if the RL is playing against a human in the game, how am I able to evaluate its performance properly?
  
  **Tags:** reinforcement-learning, rewards, performance, return, testing
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/20903/2444) you have a related question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-08-13. Score: 3 | Accepted answer
  
  When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of comparing it to humans, you might want to compare the game results of all the games played.
  
  # Reward Comparison
  
  Often Reinforcement Learning algorithms are compared by using the rewards (either direct, maximum or average in time/iteration). For example, in [this page about RL](https://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html) a comparison of two algorithms is shown:
  
  [![Comparing two RL algorithms](https://i.sstatic.net/zDk9w.png)](https://i.sstatic.net/zDk9w.png)
  
  Or, when you know the optimal actions, you can plot the number of plays/iterations against the percentage of actions. See for example this [RL comparison on the 10-armed testbed problem](http://incompleteideas.net/book/first/ebook/node22.html):
  
  ![Comparing two RL algorithms with optimal actions](https://i.sstatic.net/buIpx.png)
  
  [Henderson et al. 2017](https://arxiv.org/abs/1709.06560) have a whole section about the evaluation metrics of Reinforcement Learning algorithms. They also comment on the plotting of the average or maximum cumulative rewards, moreover they mention the *sample bootstrap* method to create confidence intervals for a better comparison. Lastly, they mention that the *significance* of the improvements of the algorithms should be calculated, using a statistical test, such the [two sample t-Test](https://stattrek.com/statistics/dictionary.aspx?definition=two-sample%20t-test). Note that you should take into account the distributions of the datasets to choose the right statistical test. An interesting article related to this is [A Hitchhiker's Guide to Statistical Comparison of Reinforcement Learning Algorithms](https://openreview.net/pdf?id=ryx0N3IaIV) of [Colas et al.](https://openreview.net/pdf?id=ryx0N3IaIV).
  
  # Comparison of plays against humans
  
  To find out how well different algorithms play against humans you should do a large number of games and compare - what you consider - important parameters, for example: did the algorithm won, the time it took to win, number of points gained, etc. These values can then be compared statistically. Note that you have to think well about the setup of these experiments since you only should change the algorithms, the other parameters should stay equal. Therefore, you should - preferably - use a large number of subjects (of different ages, sexes, etc. to cover many types of people), and try to prevent any bias; think about the order, how many games are played, location, time etc.
  
  
  ---
  
  