Body: <p>I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I can train agent A and agent B using the exact parameters and agent A beats B 800 to 200.</p>
  
  <p>I think I understand why this is happening, when training starts the model is initialized with different weights, and this leads the model to find different local max/min. </p>
  
  <p>The above makes it difficult to find optimal parameters.</p>
  
  <p>What are the strategies to reduce this variance? What parameters should I look at tweaking?</p>
  
  <p><em>More details about the environment:</em></p>
  
  <p>This is a two player game (Zombie Dice); however, in my implementation so far the agents are learning to maximize expected score on their turn, so the actions and score of the opponent is ignored.</p>
  
  <p>The variance is higher when I am using purely greedy strategy with no exploitation at all. Though it exists in both cases. I would say roughly 2/3 wins for stronger side with greedy and 3/5 with exploration out of 1000 matches.</p>
  
  <p>The environment is stochastic; I have not done many assessment runs maybe 20 or 30, it is mostly eyeballing, but the differences are fairly large; therefore, I am confident that this is not due to chance.</p>
  
  <p>I tested the models against themselves, and I get scores very close to 50/50. However, two different models trained with same parameters give results very different from 50/50. I tested this with models trained with different types of parameters and it is generally the same problem.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-13T12:43:32.950Z
Comments.Text: Is this a 2-player game that your agents are learning, or are you comparing scores on separate runs of the environment? Either way, could give a little more detail about the environment and how you are assessing agents - for instance as it is Q-learning, have you swapped to fully greedy use of Q values to set the policy, and if the environment is stochastic, how may assessment runs have you done? If this is a 2-player game, have you checked variance of A vs A and B vs B?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-14T06:47:16.390Z
Comments.Text: @NeilSlater Thank you for the suggestion, I updated my question.
Comments.UserDisplayName: Akavall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-14T06:50:07.747Z
Comments.Text: There is no need to flag edits (interested users on the site can see your revision history). The goal is to have refined question text that makes the question easy to read and understand.
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><reinforcement-learning><q-learning>
Title: High variance in performance of q-learning agents trained with same parameters
UserDisplayName: Akavall
ThreadText: # High variance in performance of q-learning agents trained with same parameters
  
  (Asked by: username_0 on 2019-05-13)
  
  I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I can train agent A and agent B using the exact parameters and agent A beats B 800 to 200.
  
  I think I understand why this is happening, when training starts the model is initialized with different weights, and this leads the model to find different local max/min.
  
  The above makes it difficult to find optimal parameters.
  
  What are the strategies to reduce this variance? What parameters should I look at tweaking?
  
  *More details about the environment:*
  
  This is a two player game (Zombie Dice); however, in my implementation so far the agents are learning to maximize expected score on their turn, so the actions and score of the opponent is ignored.
  
  The variance is higher when I am using purely greedy strategy with no exploitation at all. Though it exists in both cases. I would say roughly 2/3 wins for stronger side with greedy and 3/5 with exploration out of 1000 matches.
  
  The environment is stochastic; I have not done many assessment runs maybe 20 or 30, it is mostly eyeballing, but the differences are fairly large; therefore, I am confident that this is not due to chance.
  
  I tested the models against themselves, and I get scores very close to 50/50. However, two different models trained with same parameters give results very different from 50/50. I tested this with models trained with different types of parameters and it is generally the same problem.
  
  **Tags:** deep-learning, reinforcement-learning, q-learning
  > **Comments:**
  > **username_1**: Is this a 2-player game that your agents are learning, or are you comparing scores on separate runs of the environment? Either way, could give a little more detail about the environment and how you are assessing agents - for instance as it is Q-learning, have you swapped to fully greedy use of Q values to set the policy, and if the environment is stochastic, how may assessment runs have you done? If this is a 2-player game, have you checked variance of A vs A and B vs B?  
  > **username_0**: @NeilSlater Thank you for the suggestion, I updated my question.  
  > **username_1**: There is no need to flag edits (interested users on the site can see your revision history). The goal is to have refined question text that makes the question easy to read and understand.  
  
  ---
  
  ## 0 answers
  
  