Answers.Body: <p>The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem.</p>
  <p>There are many different initialization strategies.</p>
  <ul>
  <li>Set all values to a constant (for example, zero)</li>
  <li>Sample from a distribution, such as a normal or uniform distribution</li>
  <li>There are also some heuristic methods that seem to work very well in practice; a popular one is the so-called Glorot initializer, which is named after Xavier Glorot, who introduced them <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">here</a>. Glorot initializers also sample from distribution, but they truncate the values based on the kernel complexity.</li>
  <li>For specific types of kernels, there are other defaults that seem to perform well. See for example <a href="http://proceedings.mlr.press/v37/jozefowicz15.pdf" rel="nofollow noreferrer">this paper</a>.</li>
  </ul>
  <p>Exploring initialization strategies is something I do when my model is not able to converge (gradient problems) or when the training seems to be stuck for a long time before the loss function starts to decrease. These are signs that there might be a better initialization strategy to look for.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-23T08:45:04.920Z
Answers.LastActivityDate: 2021-09-29T13:26:42.037Z
Answers.LastEditDate: 2021-09-29T13:26:42.037Z
Answers.UserDisplayName: Mustafa Radha
Body: <p>I am currently learning about CNNs. I am confused about how filters (aka kernels) are initialized.</p>
  <p>Suppose that we have a <span class="math-container">$3 \times 3$</span> kernel. How are the values of this filter initialized before training? Do you just use <a href="http://setosa.io/ev/image-kernels/" rel="noreferrer">predefined</a> image kernels? Or are they randomly initialized, then changed with backpropagation?</p>
  
Tags: <deep-learning><convolutional-neural-networks><image-recognition><filters><weights-initialization>
Title: How are the kernels initialized in a convolutional neural network?
UserDisplayName: Inkplay_
ThreadText: # How are the kernels initialized in a convolutional neural network?
  
  (Asked by: username_0 on 2018-01-22)
  
  I am currently learning about CNNs. I am confused about how filters (aka kernels) are initialized.
  
  Suppose that we have a $3 \times 3$ kernel. How are the values of this filter initialized before training? Do you just use [predefined](http://setosa.io/ev/image-kernels/) image kernels? Or are they randomly initialized, then changed with backpropagation?
  
  **Tags:** deep-learning, convolutional-neural-networks, image-recognition, filters, weights-initialization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-01-23. Score: 5 | Accepted answer
  
  The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem.
  
  There are many different initialization strategies.
  
  * Set all values to a constant (for example, zero)
  * Sample from a distribution, such as a normal or uniform distribution
  * There are also some heuristic methods that seem to work very well in practice; a popular one is the so-called Glorot initializer, which is named after Xavier Glorot, who introduced them [here](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf). Glorot initializers also sample from distribution, but they truncate the values based on the kernel complexity.
  * For specific types of kernels, there are other defaults that seem to perform well. See for example [this paper](http://proceedings.mlr.press/v37/jozefowicz15.pdf).
  
  Exploring initialization strategies is something I do when my model is not able to converge (gradient problems) or when the training seems to be stuck for a long time before the loss function starts to decrease. These are signs that there might be a better initialization strategy to look for.
  
  
  ---
  
  