Answers.Body: <p>The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic).</p>
  <p>The weight normalisation would not be needed if the asker of the question you linked had used some kind of protection against divergence, such as using a separate target network. The normalisation may have worked for them as an alternative.</p>
  <p>DQN already has experience replay and separate target network to help with this. If you do get runaway feedback causing diverging values, you already have some hyperparameters you can change to try and fix it - size of minibatch, size of replay table, number of updates between copying to target network.</p>
  <p>So, in short, you don't really need weight normalisation in Deep RL. You might want to use it for regularisation though.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T20:09:10.497Z
Answers.LastActivityDate: 2022-06-23T20:17:10.767Z
Answers.LastEditDate: 2022-06-23T20:17:10.767Z
Answers.UserDisplayName: Neil Slater
Body: <p>I recently came across this <a href="https://stackoverflow.com/questions/23775766/how-do-you-update-the-weights-in-function-approximation-with-reinforcement-learn">SO</a> question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while using a DQN and so therefore would like to when and why is this method needed.</p>
  
Tags: <reinforcement-learning><weights><weight-normalization>
Title: Why and when do we need to normalize weights in Reinforcement Learning?
UserDisplayName: desert_ranger
ThreadText: # Why and when do we need to normalize weights in Reinforcement Learning?
  
  (Asked by: username_0 on 2022-06-23)
  
  I recently came across this [SO](https://stackoverflow.com/questions/23775766/how-do-you-update-the-weights-in-function-approximation-with-reinforcement-learn) question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while using a DQN and so therefore would like to when and why is this method needed.
  
  **Tags:** reinforcement-learning, weights, weight-normalization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-23. Score: 3 | Accepted answer
  
  The kind of divergence that the other question experienced is a common problem with deep RL and temporal difference methods (Q-learning, SARSA, or any Actor Critic).
  
  The weight normalisation would not be needed if the asker of the question you linked had used some kind of protection against divergence, such as using a separate target network. The normalisation may have worked for them as an alternative.
  
  DQN already has experience replay and separate target network to help with this. If you do get runaway feedback causing diverging values, you already have some hyperparameters you can change to try and fix it - size of minibatch, size of replay table, number of updates between copying to target network.
  
  So, in short, you don't really need weight normalisation in Deep RL. You might want to use it for regularisation though.
  
  
  ---
  
  