Answers.Body: <p>Batch Normalization should be applied between all layers and their activation functions excluding the output layer. This squishes the ranges of numbers in a better range for neural network to build appropriate sized gradients.</p>
  <p>I've not seen much use of Dropout in Deep RL because the networks are usually small and overfitting isn't as much of a problem as in supervised learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T19:55:41.863Z
Answers.Text: This is quite true, but this doesn't seem to answer the question of when to let the batch normalization layers learn their normalization, during reinforcement learning.
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-29T09:10:09.207Z
Answers.Text: I would always try it. As far as I know there is no one size fits all solution. So in general just treat using BatchNorm as a hyperparameter and test your agent with and without.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T13:58:34.437Z
Answers.Text: Hmm, I should have written "where" should batch norm be used (not "when"): the algorithm itself uses the network in 3 different places (the the original question): the question is in which of these places should batch normalization be activated.
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T14:06:23.073Z
Answers.Text: If you use BatchNorm, then use it during all phases. Only Dropout is something you turn off during evaluation.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T19:59:05.910Z
Answers.Text: Batch norm is indeed always used, but the question is really about the _activation_ of its _updates_ (i.e. training vs evaluation mode): which of the three different parts of the basic Reinforcement Learning algorithm should update the running measurement of the input's statistics? I'm not sure it matters much, but maybe this is incorrect.
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T17:03:25.543Z
Answers.Text: I'm not really sure I understand your question. It will be always updated by backprop whenever your network is updated.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-01T18:20:03.267Z
Answers.Text: … and also possibly when calling it during exploration and when computing the state value of the state reached by an action (for Bellman's equation). Hence the question.
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-02T09:41:49.497Z
Answers.Text: What exactly do you mean by 'during exploration'? In general your get_action method called in rollout should not calculate a gradient. For calculating state value you should use a target net for stability, so it's use of batchnorm depends on if your state value function uses it.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-03T10:29:06.197Z
Answers.Text: Indeed, for exploring we do not calculate a gradient, but there is the choice of whether we want to accumulate statistics for the batch normalization, hence the question (the same question also holds for training, when calculating the expected discounted reward of the state reached by an action).
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T15:59:52.893Z
Answers.Text: There is no reason to accumulate statistics during interaction with environment (this is what you call exploration phase?). This is because your experienced timesteps will be stored in some kind of memory to train on. When our agents then trains on them, batchnorm will train on the data.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T10:08:24.547Z
Answers.Text: Since the agent constantly interact with the environment, what I mean by "exploration phase" is simply the early part of training, where the agent accumulates experiences (in the "memory" you refer to--the replay buffer). When the learning takes place, batch norm will indeed learn statistics. But then why not accumulate batch norm statistics when accumulating experiences too? Since no sampling takes place, this even looks like a good idea, no?
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T13:13:48.243Z
Answers.Text: Because you would do it twice then.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-17T07:41:05.603Z
Answers.Text: … or more or less, depending on how the replay buffer is sampled. All in all, it looks like when batch norm accumulates statistics might not matter too much. One question could be whether we want the batch norm statistics to be based on the exact samples on which we learn, or be skewed towards all the samples encountered during exploration episodes. If anybody has any opinion on this, that would be interesting.
Answers.UserDisplayName: Eric O. Lebigot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-26T18:34:08.143Z
Answers.LastActivityDate: 2021-08-26T18:34:08.143Z
Answers.UserDisplayName: tnfru
Body: <p>In the vanilla version of deep Q-learning, there are three places where the Q-network is queried:</p>
  <ol>
  <li><p>When exploring.</p>
  </li>
  <li><p>When training:</p>
  <p>a. When calculating the optimal value of the state reached by an action (so as to compute a target discounted reward).</p>
  <p>b. When calculating the optimal Q-value for a given state, during training (so as to nudge the network weights and better reproduce the observed reward).</p>
  </li>
  </ol>
  <p>Now, during which steps should batch normalization and dropout be activated?</p>
  <p>I couldn't find anything through a Google search.</p>
  <p>Here are my guesses, for each step:</p>
  <ol>
  <li><p>When exploring: activate batch normalization and dropout: this lets the normalizations to be learned, and gives a chance to uncertain Q-values to be selected even if they are relatively low (because the dropout can result in a Q-value prediction higher than its average).</p>
  </li>
  <li><p>When training:</p>
  <p>a. Do not activate batch normalization and dropout for calculating the optimal state value of the state reached by an action, because we want the Bellman equation to converge faster and therefore prefer stable (optimal state value) targets.</p>
  <p>b. Activate batch normalization and dropout when calculating the Q-value of a chosen action, as this is the whole idea of dropout (we use it during training).</p>
  </li>
  </ol>
  <p>What is the common wisdom on this?</p>
  
Tags: <reinforcement-learning><q-learning><deep-rl>
Title: When to activate batch normalization and dropout in deep Q-learning?
UserDisplayName: Eric O. Lebigot
ThreadText: # When to activate batch normalization and dropout in deep Q-learning?
  
  (Asked by: username_0 on 2021-08-23)
  
  In the vanilla version of deep Q-learning, there are three places where the Q-network is queried:
  
  1. When exploring.
  2. When training:
  
     a. When calculating the optimal value of the state reached by an action (so as to compute a target discounted reward).
  
     b. When calculating the optimal Q-value for a given state, during training (so as to nudge the network weights and better reproduce the observed reward).
  
  Now, during which steps should batch normalization and dropout be activated?
  
  I couldn't find anything through a Google search.
  
  Here are my guesses, for each step:
  
  1. When exploring: activate batch normalization and dropout: this lets the normalizations to be learned, and gives a chance to uncertain Q-values to be selected even if they are relatively low (because the dropout can result in a Q-value prediction higher than its average).
  2. When training:
  
     a. Do not activate batch normalization and dropout for calculating the optimal state value of the state reached by an action, because we want the Bellman equation to converge faster and therefore prefer stable (optimal state value) targets.
  
     b. Activate batch normalization and dropout when calculating the Q-value of a chosen action, as this is the whole idea of dropout (we use it during training).
  
  What is the common wisdom on this?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-26. Score: 1
  
  Batch Normalization should be applied between all layers and their activation functions excluding the output layer. This squishes the ranges of numbers in a better range for neural network to build appropriate sized gradients.
  
  I've not seen much use of Dropout in Deep RL because the networks are usually small and overfitting isn't as much of a problem as in supervised learning.
  
  > **Comments:**
  > **username_0**: This is quite true, but this doesn't seem to answer the question of when to let the batch normalization layers learn their normalization, during reinforcement learning.  
  > **username_1**: I would always try it. As far as I know there is no one size fits all solution. So in general just treat using BatchNorm as a hyperparameter and test your agent with and without.  
  > **username_0**: Hmm, I should have written "where" should batch norm be used (not "when"): the algorithm itself uses the network in 3 different places (the the original question): the question is in which of these places should batch normalization be activated.  
  > **username_1**: If you use BatchNorm, then use it during all phases. Only Dropout is something you turn off during evaluation.  
  > **username_0**: Batch norm is indeed always used, but the question is really about the _activation_ of its _updates_ (i.e. training vs evaluation mode): which of the three different parts of the basic Reinforcement Learning algorithm should update the running measurement of the input's statistics? I'm not sure it matters much, but maybe this is incorrect.  
  > **username_1**: I'm not really sure I understand your question. It will be always updated by backprop whenever your network is updated.  
  > **username_0**: … and also possibly when calling it during exploration and when computing the state value of the state reached by an action (for Bellman's equation). Hence the question.  
  > **username_1**: What exactly do you mean by 'during exploration'? In general your get_action method called in rollout should not calculate a gradient. For calculating state value you should use a target net for stability, so it's use of batchnorm depends on if your state value function uses it.  
  > **username_0**: Indeed, for exploring we do not calculate a gradient, but there is the choice of whether we want to accumulate statistics for the batch normalization, hence the question (the same question also holds for training, when calculating the expected discounted reward of the state reached by an action).  
  > **username_1**: There is no reason to accumulate statistics during interaction with environment (this is what you call exploration phase?). This is because your experienced timesteps will be stored in some kind of memory to train on. When our agents then trains on them, batchnorm will train on the data.  
  > **username_0**: Since the agent constantly interact with the environment, what I mean by "exploration phase" is simply the early part of training, where the agent accumulates experiences (in the "memory" you refer to--the replay buffer). When the learning takes place, batch norm will indeed learn statistics. But then why not accumulate batch norm statistics when accumulating experiences too? Since no sampling takes place, this even looks like a good idea, no?  
  > **username_1**: Because you would do it twice then.  
  > **username_0**: … or more or less, depending on how the replay buffer is sampled. All in all, it looks like when batch norm accumulates statistics might not matter too much. One question could be whether we want the batch norm statistics to be based on the exact samples on which we learn, or be skewed towards all the samples encountered during exploration episodes. If anybody has any opinion on this, that would be interesting.  
  
  ---
  
  