Body: <p>I have been searching a lot about <strong>standardization/normalization of rewards and targets</strong> for the DQN algorithm. <strong>For the rewards</strong>, I now use the <a href="https://github.com/openai/gym/blob/master/gym/wrappers/normalize.py" rel="nofollow noreferrer"><strong>gym wrapper</strong></a>, which only scales but not shifts the rewards by an estimate of their standard deviation, which is updated over time.</p>
  <p>However, I <strong>can't really find something about</strong> common practice for <strong>standardization/normalization of</strong> the constructed <strong>targets</strong>. Is it done on top of reward normalization? Do we just let the agent train for a while and then take an estimate of the standard deviation of the targets and scale with the estimate then, assuming we can't use any information of the environment's dynamics?</p>
  <p>If this question is too broad to be answered in general, I'll quickly list the specifics of my environment.</p>
  <ul>
  <li>Rewards are stochastic and approximately in the range [-0.2, 0) before scaling with outliers excluded</li>
  <li>The agent receives a reward after each timestep</li>
  <li>Given a state and action, the next state is deterministic</li>
  <li>Very short time horizon -&gt; 5 timesteps per episode</li>
  <li>Action space is discrete and only consists of 11 actions</li>
  <li>With reward scaling, the q-targets are approximately in the range [-3, 0] (I saved a list of targets and plotted a histogram a couple of times during training)</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-19T17:52:00.883Z
Comments.Text: Is the NN set up for regression, with linear output in last layer, and MSE loss function? That would be default for most DQN. I don't have an answer for your main question, although my *opinion* would be that no scaling is required or likely to be useful with q targets in [-3, 0]. There are other radically different options, such as distributional return predictions, although I have only used them for very simple reward schemes (e.g. +1, 0, -1 at end of episode), no idea if they would help or hinder here.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-19T18:24:02.603Z
Comments.Text: @NeilSlater thanks for the response and your view. The NN is indeed set up for regression with the Huber loss.
Comments.UserDisplayName: Peter
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-23T22:33:56.097Z
Comments.Text: Not sure about DQN, but in actor-critic algorithms (e.g. ppo) the state is always normalized, sometimes the value function is normalized (in that case, the targets for the value function are normalized so the value function outputs a normalized prediction). Never heard of reward normalization.
Comments.UserDisplayName: Taw
Tags: <reinforcement-learning><deep-rl><dqn><rewards><reward-functions>
Title: What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
UserDisplayName: Peter
ThreadText: # What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
  
  (Asked by: username_0 on 2022-04-19)
  
  I have been searching a lot about **standardization/normalization of rewards and targets** for the DQN algorithm. **For the rewards**, I now use the [**gym wrapper**](https://github.com/openai/gym/blob/master/gym/wrappers/normalize.py), which only scales but not shifts the rewards by an estimate of their standard deviation, which is updated over time.
  
  However, I **can't really find something about** common practice for **standardization/normalization of** the constructed **targets**. Is it done on top of reward normalization? Do we just let the agent train for a while and then take an estimate of the standard deviation of the targets and scale with the estimate then, assuming we can't use any information of the environment's dynamics?
  
  If this question is too broad to be answered in general, I'll quickly list the specifics of my environment.
  
  * Rewards are stochastic and approximately in the range [-0.2, 0) before scaling with outliers excluded
  * The agent receives a reward after each timestep
  * Given a state and action, the next state is deterministic
  * Very short time horizon -> 5 timesteps per episode
  * Action space is discrete and only consists of 11 actions
  * With reward scaling, the q-targets are approximately in the range [-3, 0] (I saved a list of targets and plotted a histogram a couple of times during training)
  
  **Tags:** reinforcement-learning, deep-rl, dqn, rewards, reward-functions
  
  > **Comments:**
  > **username_1**: Is the NN set up for regression, with linear output in last layer, and MSE loss function? That would be default for most DQN. I don't have an answer for your main question, although my *opinion* would be that no scaling is required or likely to be useful with q targets in [-3, 0]. There are other radically different options, such as distributional return predictions, although I have only used them for very simple reward schemes (e.g. +1, 0, -1 at end of episode), no idea if they would help or hinder here.  
  > **username_0**: @username_1 thanks for the response and your view. The NN is indeed set up for regression with the Huber loss.  
  > **username_2**: Not sure about DQN, but in actor-critic algorithms (e.g. ppo) the state is always normalized, sometimes the value function is normalized (in that case, the targets for the value function are normalized so the value function outputs a normalized prediction). Never heard of reward normalization.  
  
  ---
  
  ## 0 answers
  
  