Answers.Body: <p>It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that.</p>
  <p>But it might cause problems later, if the values move outside the interval they were trained on. For example, it might work perfectly with a 10k account balance then unexpectedly fail with a 1m balance, because NNs do not always extrapolate well <a href="https://arxiv.org/abs/2009.11848" rel="nofollow noreferrer">Xu et al, 2020</a>.</p>
  <p>You could use &quot;current unrealized gain&quot; / &quot;current account balance&quot; instead, so it will always be in the interval [0,1].</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T22:07:06.797Z
Answers.Text: Thank you for your response. So, are you saying though, that there is *no* way to normalize this kind of information? Only clever attempts at keeping it within a reasonable range?
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T23:16:15.360Z
Answers.Text: Could you clarify precisely what you mean by normalization? It could mean scale to the range [0,1]; or scale and shift to mean 0, variance 1; or other things.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T06:13:47.220Z
Answers.Text: By normalization, I mean applying any transformation (including ones you mentioned) so that the scale/magnitude of these inputs does not vary hugely from all the other inputs (which, in my case, have been normalized by subtracting the mean and dividing by the standard deviation).
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T06:15:57.453Z
Answers.Text: I'm wondering if there is any standard method/practice of how to achieve that end with the kinds of variables that are generated directly as a result of actions the agent has taken. Any method/practice that is more standard or rigorous than a heuristic of just generally being clever to try to make sure the numbers don't get too big or small. I hope this clarifies my question.
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T10:26:11.270Z
Answers.Text: From a rigorous perspective, consider that most RL algorithms assume the environment is a Markov decision process. That means the next state is conditionally independent of all previous states, given the current state and chosen action. Including a cumulative variable like account balance or balls picked up breaks that assumption.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T10:32:17.990Z
Answers.Text: I'm not familiar with the theory or practice of RL in non-Markovian environments. If you'd like to investigate further, I did see a fairly recent paper that might be a good starting point: [Reinforcement Learning with Non-Markovian Rewards](https://arxiv.org/abs/1912.02552)
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T18:02:38.187Z
Answers.Text: @LeeReeves I don't agree with the comment that including a cumul. variable like account balance breaks the Markov assumption. If you include it in the state representation, it's right there, so Markov assumption holds. Of course it does bring the practical issue that your state space becomes really big (possibly infinite), which is bad for tabular approaches. And the issue of difficulties with normalisation (as is the topic of the question here), which is bad/challenging/annoying for RL with function approximation.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T18:47:44.480Z
Answers.Text: Thank you, you're right of course. I should be careful about answering right after waking up. Are you familiar with any standard techniques for normalizing unbounded inputs? That question is in my head now too.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-13T17:27:18.203Z
Answers.Text: @DennisSoemers I agree with your assessment about maintaining the Markov assumption. Do you agree with Taw's answer? Are there other approaches?
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T21:32:19.063Z
Answers.LastActivityDate: 2022-06-10T21:32:19.063Z
Answers.UserDisplayName: Lee Reeves
Answers.Body: <p>I'll start with the literal question in the title:</p>
  <blockquote>
  <p>Do I <strong>need</strong> to normalize all state-space variables?</p>
  </blockquote>
  <p>You don't strictly <em>need</em> to in theory. It's often really useful, or sometimes borderline necessary, in practice though. It can be useful just for faster learning, it can be important for numeric stability, or just make it much easier to tune hyperparameters (like a learning rate).</p>
  <p>If it's not clear exactly how you could go about doing the normalisation, or if you have multiple different possible solutions, ideally you'd be able to empirically validate what works best for your case by trying them all and running experiments. Maybe this will simply lead to the conclusion that not normalising at all happens to work best for your particular setup.</p>
  <hr />
  <p>I am not aware of any 100% robust, standard normalisation or standardisation approaches that work all the time when you have no prior knowledge about upper/lower bounds on the values that your variables might take (or if you do have prior knowledge that there exist no such bounds). I imagine that the online tracking of mean and standard deviation, as <a href="https://ai.stackexchange.com/a/35884/1641">suggested by Taw</a>, could work well in practice, especially if your state variables don't explode too wildly at deployment time and suddenly take very different values from what you observed at training time. On the other hand, if such extreme differences do suddenly occur, the approach won't be robust against that. Similarly, you could also track the minimum and maximum observed values, and normalise to a <span class="math-container">$[0, 1]$</span> range based on that (but in practice you might exceed that range again if after training you observe more extreme values than you did during training).</p>
  <hr />
  <p>I also really like the general idea from <a href="https://ai.stackexchange.com/a/35872/1641">Lee Reeves' answer</a> about trying to model your state variables in a different way, such that they still encode the information necessary for your agent to work well, but in a manner that has less of the annoying numeric issues. This requires domain knowledge of your particular problem though.</p>
  <blockquote>
  <p>a simple example would be like if a robot was being trained to pick up balls, and one of the observations was &quot;current number of balls picked up&quot;</p>
  </blockquote>
  <p>In this simple example, why is that observation a relevant observation? My domain knowledge says it probably isn't a relevant observation at all and should be removed. If the goal continues to be to pick up as many balls as possible, why should the robot care about how many balls it already picked up before?</p>
  <ul>
  <li>If it shouldn't care, just remove the feature instead of trying to normalise it.</li>
  <li>Maybe it could be relevant if the robot also knows how many balls were lying on the floor at the start; then, knowing how many were already picked up tells you how many are still on the floor. But the same information could be represented as a value in <span class="math-container">$[0, 1]$</span> giving you the proportion of balls already picked up... that could work better then!</li>
  </ul>
  <blockquote>
  <p>For example, I have included things like &quot;current account balance&quot; and &quot;current unrealized gain&quot; in my observation space (as I believe it's useful).</p>
  </blockquote>
  <p>For this example, it'd be good to try to think of <em>why</em> you believe such variables would be useful, and then try to again find other ways to encode it (if possible). <span class="math-container">$\frac{\text{unrealised gains}}{\text{account balance}}$</span> <a href="https://ai.stackexchange.com/a/35872/1641">as suggested by Lee Reeves</a> could be a good way. A series of binary variables that provide your agent with information about the orders of magnitude of these values could be very useful too, for example:</p>
  <ul>
  <li><span class="math-container">$b_1$</span>: is the account balance above <span class="math-container">$10$</span>?</li>
  <li><span class="math-container">$b_2$</span>: is the account balance above <span class="math-container">$100$</span>?</li>
  <li><span class="math-container">$b_3$</span>: is the account balance above <span class="math-container">$100$</span>?</li>
  <li><span class="math-container">$b_4$</span>: is the account balance above <span class="math-container">$1000$</span>?</li>
  <li><span class="math-container">$b_5$</span>: is the account balance above <span class="math-container">$10,000$</span>?</li>
  <li><span class="math-container">$b_6$</span>: is the account balance above <span class="math-container">$100,000$</span>?</li>
  </ul>
  <p>I suspect you wouldn't have to keep going too far like that, probably beyond a certain point you stop caring and your agent just knows that it's very rich and that's probably all it needs to know?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T16:06:00.097Z
Answers.Text: Thank you very much for your in-depth response. I will try all of these approaches. It's quite hard choosing an answer to accept here, as every answer adds an additional layer of perspective! After a lot of thought, I'm going to accept Taw's answer as I think it most directly answers the question in the way that I originally envisioned/expected it.
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-15T08:32:46.483Z
Answers.Text: I don't know why the so-called AI algorithms require humans to consider such complex problems, this is not artificial intelligence, it is just artificial programming.
Answers.UserDisplayName: huang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-13T18:14:17.323Z
Answers.LastActivityDate: 2022-06-13T18:14:17.323Z
Answers.UserDisplayName: Dennis Soemers
Answers.Body: <p>The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization will be dependent on the actions the agent takes; this is not unique to your problem.</p>
  <p>As for your concern of the state observations going to infinity, this will not happen. Keeping with your example, the number of balls going to infinity would require the number of timesteps to also go to infinity. In practice there will have to be some finite length of the episodes, so this won't be an issue. And note, once you start a new episode, the number of balls goes back to 0, so it's not like the normalization amount can go to infinity either.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T15:11:10.910Z
Answers.Text: Thank you for your response. I had considered a running mean/StdD, but my concern is that unless I have really long episodes (which I don't), there isn't enough time for these values to reach any kind of stability. What do you think?
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T16:15:44.017Z
Answers.Text: The length of the episode should not be important here. After you have hundreds of thousands (or millions) of transitions, the change to the mean/stdev from a single episode (which consists of perhaps a few hundred or ~thousand transitions) is fairly small.
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T16:19:11.017Z
Answers.Text: So, you're saying I would normalize *across* episodes? Not just within 1 episode? So, episode 1 ends, but the state variables of episode 2 are normalized by the mean/SD of episode 1 in addition to the timesteps taken so far in episode 2?
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T16:28:11.163Z
Answers.Text: yes. the normalization amount should be the empirical mean/stdev, computed over EVERY observed transition.
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T02:04:47.843Z
Answers.LastActivityDate: 2022-06-12T02:04:47.843Z
Answers.UserDisplayName: Taw
Body: <p>I am playing around with a DRL agent in a stock-trading environment.</p>
  <p>I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't come from the environment?</p>
  <p>For example, I have included things like &quot;current account balance&quot; and &quot;current unrealized gain&quot; in my observation space (as I believe it's useful). However, I don't know how I could normalize these values, given that they are dependent on what actions the agent took, which changes every time etc.</p>
  <p>Any feedback or advice is appreciated.</p>
  <p>Will it be detrimental if I don't normalize these values (as long as they're reasonably within the orders of magnitude of my other normalized variables)?</p>
  <p>I guess a simple example would be like if a robot was being trained to pick up balls, and one of the observations was &quot;current number of balls picked up&quot;, how would you normalize that value, given that it's just a count that could technically go to infinity?</p>
  
Tags: <reinforcement-learning><deep-rl><data-preprocessing><normalisation><algorithmic-trading>
Title: Do I need to normalize all state-space variables? If so, how?
UserDisplayName: Vladimir Belik
ThreadText: # Do I need to normalize all state-space variables? If so, how?
  
  (Asked by: username_0 on 2022-05-29)
  
  I am playing around with a DRL agent in a stock-trading environment.
  
  I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't come from the environment?
  
  For example, I have included things like "current account balance" and "current unrealized gain" in my observation space (as I believe it's useful). However, I don't know how I could normalize these values, given that they are dependent on what actions the agent took, which changes every time etc.
  
  Any feedback or advice is appreciated.
  
  Will it be detrimental if I don't normalize these values (as long as they're reasonably within the orders of magnitude of my other normalized variables)?
  
  I guess a simple example would be like if a robot was being trained to pick up balls, and one of the observations was "current number of balls picked up", how would you normalize that value, given that it's just a count that could technically go to infinity?
  
  **Tags:** reinforcement-learning, deep-rl, data-preprocessing, normalisation, algorithmic-trading
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2022-06-12. Score: 2 | Accepted answer
  
  The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization will be dependent on the actions the agent takes; this is not unique to your problem.
  
  As for your concern of the state observations going to infinity, this will not happen. Keeping with your example, the number of balls going to infinity would require the number of timesteps to also go to infinity. In practice there will have to be some finite length of the episodes, so this won't be an issue. And note, once you start a new episode, the number of balls goes back to 0, so it's not like the normalization amount can go to infinity either.
  
  > **Comments:**
  > **username_0**: Thank you for your response. I had considered a running mean/StdD, but my concern is that unless I have really long episodes (which I don't), there isn't enough time for these values to reach any kind of stability. What do you think?  
  > **username_4**: The length of the episode should not be important here. After you have hundreds of thousands (or millions) of transitions, the change to the mean/stdev from a single episode (which consists of perhaps a few hundred or ~thousand transitions) is fairly small.  
  > **username_0**: So, you're saying I would normalize *across* episodes? Not just within 1 episode? So, episode 1 ends, but the state variables of episode 2 are normalized by the mean/SD of episode 1 in addition to the timesteps taken so far in episode 2?  
  > **username_4**: yes. the normalization amount should be the empirical mean/stdev, computed over EVERY observed transition.  
  
  ---
  
  ### Answer by username_1 on 2022-06-10. Score: 2
  
  It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that.
  
  But it might cause problems later, if the values move outside the interval they were trained on. For example, it might work perfectly with a 10k account balance then unexpectedly fail with a 1m balance, because NNs do not always extrapolate well [Xu et al, 2020](https://arxiv.org/abs/2009.11848).
  
  You could use "current unrealized gain" / "current account balance" instead, so it will always be in the interval [0,1].
  
  > **Comments:**
  > **username_0**: Thank you for your response. So, are you saying though, that there is *no* way to normalize this kind of information? Only clever attempts at keeping it within a reasonable range?  
  > **username_1**: Could you clarify precisely what you mean by normalization? It could mean scale to the range [0,1]; or scale and shift to mean 0, variance 1; or other things.  
  > **username_0**: By normalization, I mean applying any transformation (including ones you mentioned) so that the scale/magnitude of these inputs does not vary hugely from all the other inputs (which, in my case, have been normalized by subtracting the mean and dividing by the standard deviation).  
  > **username_0**: I'm wondering if there is any standard method/practice of how to achieve that end with the kinds of variables that are generated directly as a result of actions the agent has taken. Any method/practice that is more standard or rigorous than a heuristic of just generally being clever to try to make sure the numbers don't get too big or small. I hope this clarifies my question.  
  > **username_1**: From a rigorous perspective, consider that most RL algorithms assume the environment is a Markov decision process. That means the next state is conditionally independent of all previous states, given the current state and chosen action. Including a cumulative variable like account balance or balls picked up breaks that assumption.  
  > **username_1**: I'm not familiar with the theory or practice of RL in non-Markovian environments. If you'd like to investigate further, I did see a fairly recent paper that might be a good starting point: [Reinforcement Learning with Non-Markovian Rewards](https://arxiv.org/abs/1912.02552)  
  > **username_2**: @LeeReeves I don't agree with the comment that including a cumul. variable like account balance breaks the Markov assumption. If you include it in the state representation, it's right there, so Markov assumption holds. Of course it does bring the practical issue that your state space becomes really big (possibly infinite), which is bad for tabular approaches. And the issue of difficulties with normalisation (as is the topic of the question here), which is bad/challenging/annoying for RL with function approximation.  
  > **username_1**: Thank you, you're right of course. I should be careful about answering right after waking up. Are you familiar with any standard techniques for normalizing unbounded inputs? That question is in my head now too.  
  > **username_0**: @DennisSoemers I agree with your assessment about maintaining the Markov assumption. Do you agree with Taw's answer? Are there other approaches?  
  
  ---
  
  ### Answer by username_2 on 2022-06-13. Score: 2
  
  I'll start with the literal question in the title:
  
  > Do I **need** to normalize all state-space variables?
  
  You don't strictly *need* to in theory. It's often really useful, or sometimes borderline necessary, in practice though. It can be useful just for faster learning, it can be important for numeric stability, or just make it much easier to tune hyperparameters (like a learning rate).
  
  If it's not clear exactly how you could go about doing the normalisation, or if you have multiple different possible solutions, ideally you'd be able to empirically validate what works best for your case by trying them all and running experiments. Maybe this will simply lead to the conclusion that not normalising at all happens to work best for your particular setup.
  
  ---
  
  I am not aware of any 100% robust, standard normalisation or standardisation approaches that work all the time when you have no prior knowledge about upper/lower bounds on the values that your variables might take (or if you do have prior knowledge that there exist no such bounds). I imagine that the online tracking of mean and standard deviation, as [suggested by Taw](https://ai.stackexchange.com/a/35884/1641), could work well in practice, especially if your state variables don't explode too wildly at deployment time and suddenly take very different values from what you observed at training time. On the other hand, if such extreme differences do suddenly occur, the approach won't be robust against that. Similarly, you could also track the minimum and maximum observed values, and normalise to a $[0, 1]$ range based on that (but in practice you might exceed that range again if after training you observe more extreme values than you did during training).
  
  ---
  
  I also really like the general idea from [Lee Reeves' answer](https://ai.stackexchange.com/a/35872/1641) about trying to model your state variables in a different way, such that they still encode the information necessary for your agent to work well, but in a manner that has less of the annoying numeric issues. This requires domain knowledge of your particular problem though.
  
  > a simple example would be like if a robot was being trained to pick up balls, and one of the observations was "current number of balls picked up"
  
  In this simple example, why is that observation a relevant observation? My domain knowledge says it probably isn't a relevant observation at all and should be removed. If the goal continues to be to pick up as many balls as possible, why should the robot care about how many balls it already picked up before?
  
  * If it shouldn't care, just remove the feature instead of trying to normalise it.
  * Maybe it could be relevant if the robot also knows how many balls were lying on the floor at the start; then, knowing how many were already picked up tells you how many are still on the floor. But the same information could be represented as a value in $[0, 1]$ giving you the proportion of balls already picked up... that could work better then!
  
  > For example, I have included things like "current account balance" and "current unrealized gain" in my observation space (as I believe it's useful).
  
  For this example, it'd be good to try to think of *why* you believe such variables would be useful, and then try to again find other ways to encode it (if possible). $\frac{\text{unrealised gains}}{\text{account balance}}$ [as suggested by Lee Reeves](https://ai.stackexchange.com/a/35872/1641) could be a good way. A series of binary variables that provide your agent with information about the orders of magnitude of these values could be very useful too, for example:
  
  * $b\_1$: is the account balance above $10$?
  * $b\_2$: is the account balance above $100$?
  * $b\_3$: is the account balance above $100$?
  * $b\_4$: is the account balance above $1000$?
  * $b\_5$: is the account balance above $10,000$?
  * $b\_6$: is the account balance above $100,000$?
  
  I suspect you wouldn't have to keep going too far like that, probably beyond a certain point you stop caring and your agent just knows that it's very rich and that's probably all it needs to know?
  
  > **Comments:**
  > **username_0**: Thank you very much for your in-depth response. I will try all of these approaches. It's quite hard choosing an answer to accept here, as every answer adds an additional layer of perspective! After a lot of thought, I'm going to accept Taw's answer as I think it most directly answers the question in the way that I originally envisioned/expected it.  
  > **username_3**: I don't know why the so-called AI algorithms require humans to consider such complex problems, this is not artificial intelligence, it is just artificial programming.  
  
  ---
  
  