Answers.Body: <p>From the stanford CNN class (<a href="http://cs231n.github.io/neural-networks-2/" rel="nofollow noreferrer">http://cs231n.github.io/neural-networks-2/</a>):</p>
  
  <blockquote>
    <p>Blockquote
    Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-05T10:48:37.990Z
Answers.LastActivityDate: 2019-09-05T10:48:37.990Z
Answers.UserDisplayName: RonsenbergVI
Body: <p>I'm initialising DNN of shape [2 inputs, 2 hiddens, 1 output] with these weights and biases:</p>
  
  <pre><code>#hidden layer
  weight1= tf.Variable(tf.random_uniform([2,2], -1, 1), 
           name="layer1");
  bias1 = tf.Variable(tf.zeros([2]), name="bias1");
  
  #output layer
  weight2 = tf.Variable(tf.random_uniform([2,1], -1, 1), 
            name="layer2");
  bias2 = tf.Variable(tf.zeros([1]), name="bias2");
  </code></pre>
  
  <p>That's what I followed some online article, however, I wonder what if I initialise bias values using <code>tf.random_uniform</code> instead of <code>tf.zeros</code>? Should I choose zero biases or random biases generically?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-05T10:13:03.927Z
Comments.Text: See [How should the bias be initialized and regularized?](https://datascience.stackexchange.com/q/17987/10640).
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><deep-learning><tensorflow><random-variable>
Title: Should the biases be zero or randomly initialised?
UserDisplayName: Dan D.
ThreadText: # Should the biases be zero or randomly initialised?
  
  (Asked by: username_0 on 2019-09-05)
  
  I'm initialising DNN of shape [2 inputs, 2 hiddens, 1 output] with these weights and biases:
  #hidden layer
  weight1= tf.Variable(tf.random_uniform([2,2], -1, 1), 
           name="layer1");
  bias1 = tf.Variable(tf.zeros([2]), name="bias1");
  
  #output layer
  weight2 = tf.Variable(tf.random_uniform([2,1], -1, 1), 
            name="layer2");
  bias2 = tf.Variable(tf.zeros([1]), name="bias2");
  
  That's what I followed some online article, however, I wonder what if I initialise bias values using tf.random_uniform instead of tf.zeros? Should I choose zero biases or random biases generically?
  
  **Tags:** neural-networks, machine-learning, deep-learning, tensorflow, random-variable
  > **Comments:**
  > **username_1**: See [How should the bias be initialized and regularized?](https://datascience.stackexchange.com/q/17987/10640).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-05. Score: 3 | Accepted answer
  
  From the stanford CNN class (<http://cs231n.github.io/neural-networks-2/>):
  
  > Blockquote
  > Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.
  
  
  ---
  
  