Answers.Body: <p>The use of normalisation in neural networks and many other (but not all - decision trees are a notable exception) machine learning methods, is to improve the quality of the parameter space with respect to optimisers that will apply to it.</p>
  <p>If you are using a function approximator that benefits from normalisation in supervised learning scenarios, it will also benefit from it in reinforcement learning scenarios. That is definitely the case for neural networks. And neural networks are by far the most common approximator used in deep reinforcement learning.</p>
  <p>Unlike supervised learning, you will not have a definitive dataset where you can find mean and standard deviation in order to scale to the common <span class="math-container">$\mu = 0, \sigma = 1$</span>. Instead you will want to scale to a range, such as <span class="math-container">$[-1, 1]$</span>.</p>
  <p>You may also want to perform some basic feature engineering first, such as using log of a value, or some power of it - anything that would make the distribution of values you expect to see more like a Normal distribution. Again, this is something you could do in supervised learning more easily, but maybe you know enough about the feature to make a good guess.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-08T18:22:31.727Z
Answers.LastActivityDate: 2021-09-08T18:22:31.727Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>On creating custom environments:</p>
  <blockquote>
  <p>... always normalize your observation space when you can, i.e., when you know the boundaries
  (From <a href="https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html#tips-and-tricks-when-creating-a-custom-environment" rel="nofollow noreferrer">stable-baselines</a>)</p>
  </blockquote>
  <p>You could normalize them as part of the environment's state space or before passing them as input to the policy. Depending on the the agent's algorithm implementation, what works for you may vary.</p>
  <p>(See <a href="https://stats.stackexchange.com/a/347644">this answer</a> from a related question)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-09T06:37:09.817Z
Answers.LastActivityDate: 2021-10-09T06:37:09.817Z
Answers.UserDisplayName: mugoh
Body: <p>I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning.</p>
  <p>In my custom environment, the state/observation values are in a different range. For example, one observation is in the range of [1, 20], while another is in [0, 50000]. Should I apply normalization or not? I am confused. Any suggestions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-07T08:33:01.280Z
Comments.Text: At the very least you’ll have to train separate networks for the different size states.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T01:50:17.747Z
Comments.Text: is it the same field of the state (observation), or are they two different fields of the state that have different magnitude and you think need normalization across fields? If it is the same field, then doesn't it (always) have the same range (not [1,20] but always [min,max] where min/max is the known min/max in an episode) ? Or do you mean that the states' magnitude changes on different episodes?
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T04:18:00.533Z
Comments.Text: They are different fields. The observation is of dimension (5 x 1). One of them is in the range of [1, 20], other is in [0, 50000]. Actually, the first one (1-20) is like time steps which increase with the number of steps in each episode, whereas the second one (0 -50000) is like a residual budget. It decreases with steps and if it becomes zero, the agent is penalized.
Comments.UserDisplayName: moyukh
Tags: <reinforcement-learning><deep-rl><regularization><normalisation><observation-spaces>
Title: Should I apply normalization to the observations in deep reinforcement learning?
UserDisplayName: moyukh
ThreadText: # Should I apply normalization to the observations in deep reinforcement learning?
  
  (Asked by: username_0 on 2021-09-07)
  
  I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning.
  
  In my custom environment, the state/observation values are in a different range. For example, one observation is in the range of [1, 20], while another is in [0, 50000]. Should I apply normalization or not? I am confused. Any suggestions?
  
  **Tags:** reinforcement-learning, deep-rl, regularization, normalisation, observation-spaces
  > **Comments:**
  > **username_1**: At the very least you’ll have to train separate networks for the different size states.  
  > **username_2**: is it the same field of the state (observation), or are they two different fields of the state that have different magnitude and you think need normalization across fields? If it is the same field, then doesn't it (always) have the same range (not [1,20] but always [min,max] where min/max is the known min/max in an episode) ? Or do you mean that the states' magnitude changes on different episodes?  
  > **username_0**: They are different fields. The observation is of dimension (5 x 1). One of them is in the range of [1, 20], other is in [0, 50000]. Actually, the first one (1-20) is like time steps which increase with the number of steps in each episode, whereas the second one (0 -50000) is like a residual budget. It decreases with steps and if it becomes zero, the agent is penalized.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-09-08. Score: 3
  
  The use of normalisation in neural networks and many other (but not all - decision trees are a notable exception) machine learning methods, is to improve the quality of the parameter space with respect to optimisers that will apply to it.
  
  If you are using a function approximator that benefits from normalisation in supervised learning scenarios, it will also benefit from it in reinforcement learning scenarios. That is definitely the case for neural networks. And neural networks are by far the most common approximator used in deep reinforcement learning.
  
  Unlike supervised learning, you will not have a definitive dataset where you can find mean and standard deviation in order to scale to the common $\mu = 0, \sigma = 1$. Instead you will want to scale to a range, such as $[-1, 1]$.
  
  You may also want to perform some basic feature engineering first, such as using log of a value, or some power of it - anything that would make the distribution of values you expect to see more like a Normal distribution. Again, this is something you could do in supervised learning more easily, but maybe you know enough about the feature to make a good guess.
  
  
  ---
  
  ### Answer by username_4 on 2021-10-09. Score: 2
  
  On creating custom environments:
  
  > ... always normalize your observation space when you can, i.e., when you know the boundaries
  > (From [stable-baselines](https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html#tips-and-tricks-when-creating-a-custom-environment))
  
  You could normalize them as part of the environment's state space or before passing them as input to the policy. Depending on the the agent's algorithm implementation, what works for you may vary.
  
  (See [this answer](https://stats.stackexchange.com/a/347644) from a related question)
  
  
  ---
  
  