Answers.Body: <p>The idea of batchnorm is to normalize the input to a range that is suitable for subsequent computation (say tanh activation). The most naive idea is to normalize each sample by subtracting the batch mean and then dividing it by the batch's standard derivation. So the overall normalized batch will have zero mean and unit variance.</p>
  <p>However, how do we know that zero mean and unit variance is the best for subsequent operations? We don't.</p>
  <p>That is why it also introduces beta and gamma to readjust the mean to beta and standard derivation gamma. But these parameters will be learned instead of preset. gamma is initialized to 1 to begin with. But if you are recovering a checkpoint or transferring a model, you may want to initialize it to the value from the trained model.</p>
  <p>For practical purposes, I believe that should be automatically taken care of when you reload a checkpoint with tensorflow and so you should not need to worry about it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-12T18:31:29.243Z
Answers.Text: My Question was more around how to set the Gamma initializer as in the documents ,(I knoew what is Batch normalization). nothing was said about the accepatble answers , So as I have figuered out the only argumnets it takes is the name of its regularization techniques it may take.
  Currently Tf (2.11) it only accepts : 'L1L2' 'L1' 'L2'
Answers.UserDisplayName: sara yaghoobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-10T18:58:45.400Z
Answers.LastActivityDate: 2023-02-10T18:58:45.400Z
Answers.UserDisplayName: user559678
Body: <p>I have read tensorflow's documents about batch normzalization , but still don't get what is the gamma regulizer?
  the link to document:
  <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-11T11:22:21.443Z
Comments.Text: You're asking 2 questions here: 1. how to set gamma, 2. what is gamma. Please, ask only one question per post and explain why you don't understand what gamma is or how to set  it.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-12T18:29:11.767Z
Comments.Text: No actually I was only asking for the exact paramet of "gamma regulizer" and how should this be initialized
Comments.UserDisplayName: sara yaghoobi
Tags: <deep-learning><tensorflow><hyperparameter-optimization>
Title: L2 regularization in BN layers, how to set gamma?
UserDisplayName: sara yaghoobi
ThreadText: # L2 regularization in BN layers, how to set gamma?
  
  (Asked by: username_0 on 2023-02-09)
  
  I have read tensorflow's documents about batch normzalization , but still don't get what is the gamma regulizer?
  the link to document:
  <https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization>
  
  **Tags:** deep-learning, tensorflow, hyperparameter-optimization
  
  > **Comments:**
  > **username_1**: You're asking 2 questions here: 1. how to set gamma, 2. what is gamma. Please, ask only one question per post and explain why you don't understand what gamma is or how to set  it.  
  > **username_0**: No actually I was only asking for the exact paramet of "gamma regulizer" and how should this be initialized  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-10. Score: 0
  
  The idea of batchnorm is to normalize the input to a range that is suitable for subsequent computation (say tanh activation). The most naive idea is to normalize each sample by subtracting the batch mean and then dividing it by the batch's standard derivation. So the overall normalized batch will have zero mean and unit variance.
  
  However, how do we know that zero mean and unit variance is the best for subsequent operations? We don't.
  
  That is why it also introduces beta and gamma to readjust the mean to beta and standard derivation gamma. But these parameters will be learned instead of preset. gamma is initialized to 1 to begin with. But if you are recovering a checkpoint or transferring a model, you may want to initialize it to the value from the trained model.
  
  For practical purposes, I believe that should be automatically taken care of when you reload a checkpoint with tensorflow and so you should not need to worry about it.
  
  > **Comments:**
  > **username_0**: My Question was more around how to set the Gamma initializer as in the documents ,(I knoew what is Batch normalization). nothing was said about the accepatble answers , So as I have figuered out the only argumnets it takes is the name of its regularization techniques it may take. Currently Tf (2.11) it only accepts : 'L1L2' 'L1' 'L2'  
  
  ---
  
  