Answers.Body: <p>Notably, these two tips/tricks are useful because we are assuming the context of <strong>deep</strong> reinforcement learning here, as you pointed out. In DRL, the RL algorithm is guided in some fashion by a deep neural network, and the reasons for normalizing stem from the gradient descent algorithm and the architecture of the network.</p>
  
  <blockquote>
    <p>How does this affect training?</p>
  </blockquote>
  
  <p>An observation from the observation space is often used as an input to a neural network in DRL algorithms, and normalizing the input to neural networks is beneficial for many reasons (e.g. increases convergence speed, aids computer precision, prevents divergence of parameters, allows for easier hyperparameter tuning, etc.). These are standard results in DL theory and practice, so I won't provide details here.</p>
  
  <blockquote>
    <p>And more specifically, why on continuous action spaces we need to normalize also the action's values?</p>
  </blockquote>
  
  <p>Most popular discrete action space DRL algorithms (e.g. DQN) have one output node for each possible action in the neural net. The value of the output node may be a q-value (value-based algorithm) or a probability of taking that action (policy-based algorithm). </p>
  
  <p>In contrast, a continuous action space DRL algorithm simply cannot have an output node for each possible action, as the action space is continuous. The output is usually the actual action to be taken by the agent or some parameters that could be used to construct the action (e.g. PPO outputs a mean and standard deviation and then an action is sampled from the corresponding Gaussian distribution - this phenomenon is mentioned in your linked reference). Therefore, normalizing the action space of a DRL algorithm is analogous to normalizing the outputs of the corresponding neural network, which is known to increase training speed and prevent divergence. Again, a quick search will yield some good resources if you are interested in these results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-05T16:53:01.207Z
Answers.LastActivityDate: 2020-06-05T16:53:01.207Z
Answers.UserDisplayName: DeepQZero
Body: <p>I was reading <a href="https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html#tips-and-tricks-when-creating-a-custom-environment" rel="nofollow noreferrer">here</a> tips &amp; tricks for training in DRL and I noticed the following:</p>
  
  <blockquote>
    <ul>
    <li>always normalize your observation space when you can, i.e., when you know the boundaries</li>
    <li>normalize your action space and make it symmetric when continuous (cf potential issue below) A good practice is to rescale your actions to lie in [-1, 1]. This does not limit you as you can easily rescale the action inside the environment</li>
    </ul>
  </blockquote>
  
  <p>I am working on a discrete action space but it is quite difficult to normalize my states when I don't actually know the full range for each feature (only an estimation).</p>
  
  <p>How does this affect training? And more specifically, why on continuous action spaces we need to normalize also the action's values?</p>
  
Tags: <reinforcement-learning><training><deep-rl><a3c>
Title: Why do we also need to normalize the action's values on continuous action spaces?
UserDisplayName: mkanakis
ThreadText: # Why do we also need to normalize the action's values on continuous action spaces?
  
  (Asked by: username_0 on 2020-05-26)
  
  I was reading [here](https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html#tips-and-tricks-when-creating-a-custom-environment) tips & tricks for training in DRL and I noticed the following:
  
  > * always normalize your observation space when you can, i.e., when you know the boundaries
  > * normalize your action space and make it symmetric when continuous (cf potential issue below) A good practice is to rescale your actions to lie in [-1, 1]. This does not limit you as you can easily rescale the action inside the environment
  
  I am working on a discrete action space but it is quite difficult to normalize my states when I don't actually know the full range for each feature (only an estimation).
  
  How does this affect training? And more specifically, why on continuous action spaces we need to normalize also the action's values?
  
  **Tags:** reinforcement-learning, training, deep-rl, a3c
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-05. Score: 3 | Accepted answer
  
  Notably, these two tips/tricks are useful because we are assuming the context of **deep** reinforcement learning here, as you pointed out. In DRL, the RL algorithm is guided in some fashion by a deep neural network, and the reasons for normalizing stem from the gradient descent algorithm and the architecture of the network.
  
  > How does this affect training?
  
  An observation from the observation space is often used as an input to a neural network in DRL algorithms, and normalizing the input to neural networks is beneficial for many reasons (e.g. increases convergence speed, aids computer precision, prevents divergence of parameters, allows for easier hyperparameter tuning, etc.). These are standard results in DL theory and practice, so I won't provide details here.
  
  > And more specifically, why on continuous action spaces we need to normalize also the action's values?
  
  Most popular discrete action space DRL algorithms (e.g. DQN) have one output node for each possible action in the neural net. The value of the output node may be a q-value (value-based algorithm) or a probability of taking that action (policy-based algorithm).
  
  In contrast, a continuous action space DRL algorithm simply cannot have an output node for each possible action, as the action space is continuous. The output is usually the actual action to be taken by the agent or some parameters that could be used to construct the action (e.g. PPO outputs a mean and standard deviation and then an action is sampled from the corresponding Gaussian distribution - this phenomenon is mentioned in your linked reference). Therefore, normalizing the action space of a DRL algorithm is analogous to normalizing the outputs of the corresponding neural network, which is known to increase training speed and prevent divergence. Again, a quick search will yield some good resources if you are interested in these results.
  
  
  ---
  
  