Answers.Body: <p>(I'll provide a couple of ideas, but I don't think these would fully address the problem.)</p>
  <p>If you normalize with running statistics (e.g. mean and std) as in the online case, you would get two different results for the same input, say <span class="math-container">$s$</span>, because it would be normalized with different statistics. I think this would introduce some <em>non-stationarity</em> in the problem, which one usually wants to avoid.</p>
  <p>One possible solution is to also store the statistics computed at timestep <span class="math-container">$t$</span>, say <span class="math-container">$\mu_t$</span> and <span class="math-container">$\sigma_t$</span> such that when you replay a transition <span class="math-container">$(s_t, a_t, r_t, s'_t, \mu_t,\sigma_t)$</span> you also have the correct statistics for that timestep: which should avoid the distribution shift.</p>
  <p>In general, if we assume that the distribution of <span class="math-container">$s_t$</span> is stationary i.e. it doesn't change with time, then, in the limit <span class="math-container">$t\to\infty$</span> (also assuming unbiased estimators), the statistics <span class="math-container">$\mu_t$</span> and <span class="math-container">$\sigma_t$</span> will converge to the true mean and std. If so, i.e. when you detect that <span class="math-container">$\mu_t$</span> is close to <span class="math-container">$\mu_{t+1}$</span> then you can stop storing the stats for each <span class="math-container">$t$</span> and only use the latest ones to avoid the extra memory cost. But storing that especially at the beginning of training can help stabilize the agent.</p>
  <p>Alternatively, you can consider <span class="math-container">$K$</span> steps of interaction and keep the stats constant for that number of steps. Then you can either recompute them considering the new <span class="math-container">$K$</span> transitions or flush the replay buffer (which probably has the same effect, but reduces sample-efficiency.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T18:26:21.927Z
Answers.LastActivityDate: 2023-07-12T18:26:21.927Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the network, to train it. If one doesn't have a standard way of normalizing, this would lead to a difference in the distribution between the inputs for prediction and training, despite having the same values.</p>
  <p>My questions follows from the question that I asked over <a href="https://ai.stackexchange.com/questions/41216/how-does-one-normalize-observations-in-online-reinforcement-learning/41223#41223">here</a> (I have been told to use a new post for each question). The accepted answer suggested computing running statistics for my inputs. But I doubt that'll work with the problem described above.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-12T22:08:35.213Z
Comments.Text: that's pretty much like having a batch normalization layer at the beginning of the network... BN is known to be somewhat difficult to tune, however usually you would know some statistics about the data you are feeding the network, you can use those to normalize/standardize
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning><deep-rl><dqn><off-policy-methods>
Title: How would one normalize observations in off-policy online reinforcement learning?
UserDisplayName: desert_ranger
ThreadText: # How would one normalize observations in off-policy online reinforcement learning?
  
  (Asked by: username_0 on 2023-07-10)
  
  In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the network, to train it. If one doesn't have a standard way of normalizing, this would lead to a difference in the distribution between the inputs for prediction and training, despite having the same values.
  
  My questions follows from the question that I asked over [here](https://ai.stackexchange.com/questions/41216/how-does-one-normalize-observations-in-online-reinforcement-learning/41223#41223) (I have been told to use a new post for each question). The accepted answer suggested computing running statistics for my inputs. But I doubt that'll work with the problem described above.
  
  **Tags:** reinforcement-learning, deep-rl, dqn, off-policy-methods
  
  > **Comments:**
  > **username_1**: that's pretty much like having a batch normalization layer at the beginning of the network... BN is known to be somewhat difficult to tune, however usually you would know some statistics about the data you are feeding the network, you can use those to normalize/standardize  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-12. Score: 1 | Accepted answer
  
  (I'll provide a couple of ideas, but I don't think these would fully address the problem.)
  
  If you normalize with running statistics (e.g. mean and std) as in the online case, you would get two different results for the same input, say $s$, because it would be normalized with different statistics. I think this would introduce some *non-stationarity* in the problem, which one usually wants to avoid.
  
  One possible solution is to also store the statistics computed at timestep $t$, say $\mu\_t$ and $\sigma\_t$ such that when you replay a transition $(s\_t, a\_t, r\_t, s'\_t, \mu\_t,\sigma\_t)$ you also have the correct statistics for that timestep: which should avoid the distribution shift.
  
  In general, if we assume that the distribution of $s\_t$ is stationary i.e. it doesn't change with time, then, in the limit $t\to\infty$ (also assuming unbiased estimators), the statistics $\mu\_t$ and $\sigma\_t$ will converge to the true mean and std. If so, i.e. when you detect that $\mu\_t$ is close to $\mu\_{t+1}$ then you can stop storing the stats for each $t$ and only use the latest ones to avoid the extra memory cost. But storing that especially at the beginning of training can help stabilize the agent.
  
  Alternatively, you can consider $K$ steps of interaction and keep the stats constant for that number of steps. Then you can either recompute them considering the new $K$ transitions or flush the replay buffer (which probably has the same effect, but reduces sample-efficiency.)
  
  
  ---
  
  