Answers.Body: <p>You may be interested in section 3.2 of this paper <a href="https://arxiv.org/pdf/2006.05990.pdf" rel="nofollow noreferrer">What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study</a> (2020) by Google Research.</p>
  <p>They claim that the initialization of the policy is very important to performance, sometimes making a huge (66%) improvement, just from the initialization of the policy.</p>
  <p>I'm assuming you already know that the kernel weights in neural networks are initialized using the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotUniform" rel="nofollow noreferrer">glorot uniform</a>  distribution; in that paper, they suggest that for the output layer, those weights should additionally be divided by 100.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T15:55:34.070Z
Answers.LastActivityDate: 2022-01-25T23:01:26.500Z
Answers.LastEditDate: 2022-01-25T23:01:26.500Z
Answers.UserDisplayName: Taw
Body: <p>I would like to <strong>train a neural network</strong> (NN) so that it <strong>learns the policy and value function</strong> for my agent.</p>
  <p>Since I am using reinforcement learning and do not want to prefer certain actions in certain states at the beginning of the learning, ideally, my <strong>NN</strong> should be <strong>initialized</strong> in a way that it <strong>predicts a uniform policy</strong> for all of the actions in every state and then during training, it will adjust its weights based on the observations.</p>
  <p>The idea for this weight initialization is to <strong>speed up the training process</strong> by not &quot;delearning&quot; random initial policy that can be off by quite some margin and to also guarantee equal exploration from every state straight from the beginning.</p>
  <p>I would like to ask two questions about this topic:</p>
  <ul>
  <li>Is this a good idea?</li>
  <li>Are there any available tools for achieving this?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T15:25:25.570Z
Comments.Text: what you describe sounds like a simple uniform weight initialization, which all deep learning libraries allow out of the shelf. Be aware though that initializing a network to output uniform stuff doesn't necessary mean faster training. It's an assumption that sounds logical to us but it's not correlated at all with the optimization problem math the network is suppose to learn.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T16:11:48.203Z
Comments.Text: Can you please put your **specific** question in the title? "Weight initialization for Deep Reinforcement Learning" is not a question and it's quite general.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T17:18:47.830Z
Comments.Text: @EdoardoGuerriero uniform weight initialization in the last layer does not necessarily produce uniform policy, since final output also depends on the values coming from the previous layers, or am I thinking somewhere wrong? (e.g. if we would have same weights in last layer but use ReLU in the penultimate layer then final probabilities will be scaled by the input coming from the ReLU, won't they?)
Comments.UserDisplayName: Druudik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T17:38:55.567Z
Comments.Text: @Druudik that is true, there will always be some unbalance due to the input values, but still uniform initialization would be the only one capable to minimize that variation. There's no way to generically account for input variation to output a constant value, or to say it differently, you would have to train the model on that task first. And that would be tricky as well, for example it's not hard to see that if using biases the model would probably converge immediately on a solution with zero weights an 1/n bias values. (n = amount f output nodes)
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T17:51:24.353Z
Comments.Text: Thanks. The title was not really in the form of a question. Check it now. Is that your question? By the way, for completeness, it might be a good idea to specify also the RL algorithm you're using.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T18:06:19.050Z
Comments.Text: @nbro yes that is exactly what I was trying to ask! Regarding the RL algorithm, I meant it as a more of a general question that can be applied to any RL algorithm which uses NN as a model for value and policy function.
Comments.UserDisplayName: Druudik
Tags: <reinforcement-learning><deep-learning><deep-rl><weights-initialization>
Title: How should I initialize the weights of the neural network so that the initial policy is uniform?
UserDisplayName: Druudik
ThreadText: # How should I initialize the weights of the neural network so that the initial policy is uniform?
  
  (Asked by: username_0 on 2022-01-24)
  
  I would like to **train a neural network** (NN) so that it **learns the policy and value function** for my agent.
  
  Since I am using reinforcement learning and do not want to prefer certain actions in certain states at the beginning of the learning, ideally, my **NN** should be **initialized** in a way that it **predicts a uniform policy** for all of the actions in every state and then during training, it will adjust its weights based on the observations.
  
  The idea for this weight initialization is to **speed up the training process** by not "delearning" random initial policy that can be off by quite some margin and to also guarantee equal exploration from every state straight from the beginning.
  
  I would like to ask two questions about this topic:
  
  * Is this a good idea?
  * Are there any available tools for achieving this?
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, weights-initialization
  
  > **Comments:**
  > **username_1**: what you describe sounds like a simple uniform weight initialization, which all deep learning libraries allow out of the shelf. Be aware though that initializing a network to output uniform stuff doesn't necessary mean faster training. It's an assumption that sounds logical to us but it's not correlated at all with the optimization problem math the network is suppose to learn.  
  > **username_2**: Can you please put your **specific** question in the title? "Weight initialization for Deep Reinforcement Learning" is not a question and it's quite general.  
  > **username_0**: @username_1 uniform weight initialization in the last layer does not necessarily produce uniform policy, since final output also depends on the values coming from the previous layers, or am I thinking somewhere wrong? (e.g. if we would have same weights in last layer but use ReLU in the penultimate layer then final probabilities will be scaled by the input coming from the ReLU, won't they?)  
  > **username_1**: @username_0 that is true, there will always be some unbalance due to the input values, but still uniform initialization would be the only one capable to minimize that variation. There's no way to generically account for input variation to output a constant value, or to say it differently, you would have to train the model on that task first. And that would be tricky as well, for example it's not hard to see that if using biases the model would probably converge immediately on a solution with zero weights an 1/n bias values. (n = amount f output nodes)  
  > **username_2**: Thanks. The title was not really in the form of a question. Check it now. Is that your question? By the way, for completeness, it might be a good idea to specify also the RL algorithm you're using.  
  > **username_0**: @username_2 yes that is exactly what I was trying to ask! Regarding the RL algorithm, I meant it as a more of a general question that can be applied to any RL algorithm which uses NN as a model for value and policy function.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-01-24. Score: 3 | Accepted answer
  
  You may be interested in section 3.2 of this paper [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://arxiv.org/pdf/2006.05990.pdf) (2020) by Google Research.
  
  They claim that the initialization of the policy is very important to performance, sometimes making a huge (66%) improvement, just from the initialization of the policy.
  
  I'm assuming you already know that the kernel weights in neural networks are initialized using the [glorot uniform](https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotUniform) distribution; in that paper, they suggest that for the output layer, those weights should additionally be divided by 100.
  
  
  ---
  
  