Answers.Body: <p>Alternatively you can compute a <em>running mean</em>, <span class="math-container">$\mu_t$</span>, and std, <span class="math-container">$\sigma_t$</span>, of your online data <span class="math-container">$x_t$</span>, and then standardize at each timestep, <span class="math-container">$t$</span>:
  <span class="math-container">$$\begin{align}
  \mu_t &amp;\leftarrow \mu_{t+1} + \frac{(x_t - \mu_{t-1})}{n}\\
  n &amp;\leftarrow n + 1 \\
  \sigma_t &amp;\leftarrow \sigma_{t-1} + (x_t - \mu_t) * (x_t - \mu_{t-1})
  \end{align}$$</span></p>
  <p>Initially you would have: <span class="math-container">$\mu_0=0$</span>, <span class="math-container">$\sigma_0=1$</span>, and <span class="math-container">$n=1$</span>.</p>
  <p>For reference see: <a href="https://dsp.stackexchange.com/questions/811/determining-the-mean-and-standard-deviation-in-real-time">1</a> and <a href="https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance" rel="nofollow noreferrer">2</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-10T11:24:03.320Z
Answers.LastActivityDate: 2023-07-10T11:24:03.320Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its maximum value before inputting it into a policy. However, I feel that doing so, could change the distribution of the data. For instance, let's say we have 2 inputs that we normalize on the fly - [5, 15, 20] and [10, 20, 30] to [0.25, .75, 1] and [0.33, .66 and 1]. Now let's say the true maximum value of the environment is 100. Then the true normalized values should have been [0.05, .15, .20] and [0.1, .20, 0.3]. Wouldn't this adversely affect learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-10T03:20:12.527Z
Comments.Text: See https://stats.stackexchange.com/questions/261940/how-do-i-normalize-data-in-online-learning
Comments.UserDisplayName: Brian O'Donnell
Tags: <reinforcement-learning><deep-rl><model-free-methods>
Title: How does one normalize observations in online reinforcement learning
UserDisplayName: desert_ranger
ThreadText: # How does one normalize observations in online reinforcement learning
  
  (Asked by: username_0 on 2023-07-09)
  
  I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its maximum value before inputting it into a policy. However, I feel that doing so, could change the distribution of the data. For instance, let's say we have 2 inputs that we normalize on the fly - [5, 15, 20] and [10, 20, 30] to [0.25, .75, 1] and [0.33, .66 and 1]. Now let's say the true maximum value of the environment is 100. Then the true normalized values should have been [0.05, .15, .20] and [0.1, .20, 0.3]. Wouldn't this adversely affect learning?
  
  **Tags:** reinforcement-learning, deep-rl, model-free-methods
  
  > **Comments:**
  > **username_1**: See https://stats.stackexchange.com/questions/261940/how-do-i-normalize-data-in-online-learning  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-10. Score: 3 | Accepted answer
  
  Alternatively you can compute a *running mean*, $\mu\_t$, and std, $\sigma\_t$, of your online data $x\_t$, and then standardize at each timestep, $t$:
  $$\begin{align}
  \mu\_t &\leftarrow \mu\_{t+1} + \frac{(x\_t - \mu\_{t-1})}{n}\\
  n &\leftarrow n + 1 \\
  \sigma\_t &\leftarrow \sigma\_{t-1} + (x\_t - \mu\_t) \* (x\_t - \mu\_{t-1})
  \end{align}$$
  
  Initially you would have: $\mu\_0=0$, $\sigma\_0=1$, and $n=1$.
  
  For reference see: [1](https://dsp.stackexchange.com/questions/811/determining-the-mean-and-standard-deviation-in-real-time) and [2](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance).
  
  
  ---
  
  