Answers.Body: <p>If you're using a discount factor less than 1, you should be able to compute a maximum return (likewise, a minimum return) based on the max (min) reward you can earn at each timestep. However, this issue you bring up is usually cited as a difficulty with C51. I think people tend to simply use fixed values for the min/max return (or just make rough estimates). If you want to avoid this, I recommend looking into the QR-DQN algorithm which circumvents this issue altogether and is more theoretically sound.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T14:49:55.860Z
Answers.LastActivityDate: 2020-08-28T14:49:55.860Z
Answers.UserDisplayName: harwiltz
Body: <p><strong>Background:</strong> In C51 DQNs you must specify a v-min/max to be used during training. The way this is generally done is you take the max score possible for the game and set that to v-max, then v-min is just negative v-max. For a game like Pong deciding the v-min/max is simple because the max score possible is 20, therefore, <code>v_min=-20</code> and <code>v_max=20</code>.</p>
  <p><strong>Question:</strong> In a game like Space Invaders, there is no max score, so how would I calculate the v-min/max for a C51 DQN?</p>
  
Tags: <reinforcement-learning><dqn><probability-distribution>
Title: How to calculate v min and v max for C51 DQN
UserDisplayName: Leon
ThreadText: # How to calculate v min and v max for C51 DQN
  
  (Asked by: username_0 on 2020-08-26)
  
  **Background:** In C51 DQNs you must specify a v-min/max to be used during training. The way this is generally done is you take the max score possible for the game and set that to v-max, then v-min is just negative v-max. For a game like Pong deciding the v-min/max is simple because the max score possible is 20, therefore, `v_min=-20` and `v_max=20`.
  
  **Question:** In a game like Space Invaders, there is no max score, so how would I calculate the v-min/max for a C51 DQN?
  
  **Tags:** reinforcement-learning, dqn, probability-distribution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-28. Score: 3 | Accepted answer
  
  If you're using a discount factor less than 1, you should be able to compute a maximum return (likewise, a minimum return) based on the max (min) reward you can earn at each timestep. However, this issue you bring up is usually cited as a difficulty with C51. I think people tend to simply use fixed values for the min/max return (or just make rough estimates). If you want to avoid this, I recommend looking into the QR-DQN algorithm which circumvents this issue altogether and is more theoretically sound.
  
  
  ---
  
  