Answers.Body: <p>It is usual to initialize parameters with &quot;a good guess&quot;, when you have prior information, in order to help the model converging.</p>
  <p>However in deep learning most of the time you have no clue what the weights should be, so you initialize them randomly following certain schedules like <strong>Xavier initialization</strong> or <strong>Kaiming</strong>, that prevent gradients from vanishing or exploding (due to a bad initialization) by setting the mean activations to be zero and the variance of the activations to be constant accross all layers.</p>
  <p>Regarding your questions :</p>
  <ol>
  <li><p>Yes it is totally possible that multiples filters learn the same thing. In fact, a good part of the filters may be completly useless, which is the idea behing <strong>network pruning</strong> that tries to delete neurons in a network while keeping the same performances.</p>
  </li>
  <li><p>Yes, the network you'll obtain after training will depend on the initialization. You can have different results depending on the initialization (just permuting the filters, for instance, give a new network with same capabilities), since anyway you will never get the global optimum for your task but only sub-optimal weights. However if trained properly, the final results should not very too much.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-08T10:31:51.157Z
Answers.LastActivityDate: 2024-02-08T10:31:51.157Z
Answers.UserDisplayName: Lelouch
Body: <p>I learned that when CNN filters are defined, they are initialized with random weights and bias(Im not sure about bias).</p>
  <p>Then as learning step goes on, the weight values change and each filter makes its own feature map.</p>
  <p>What I don't understand is that, if filter is initialized with random values, is there any chance that</p>
  <ol>
  <li>different filters make the same feature map</li>
  <li>feature map varies every time it repeats.</li>
  </ol>
  <p>It seems little unefficient to initialize the weights of every filter randomly. More precisely, I think (in most of the networks) the number of filters is too small to get meaningful features.</p>
  <p>Is the second case is why CNN has randomness?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T11:06:42.567Z
Comments.Text: first of all its is best to initialize randomly according to some distribution, for instance if you are using Relu activation, initialize according He-normal, or He-uniform, see the corresponding papers for why. second, if you initialize all weights similarly well then they all be treated in the same way by the gradient descent, and converge to similar values. finally in stochastic optimization, you need some kind to diversity to be able to explore the search space and find the solution. You do not no where the optimum solution is you need to find it, diversity ensures that you have a chance.
Comments.UserDisplayName: Reza_va
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-26T11:10:15.837Z
Comments.Text: Can you please put your **specific question** in the title? Thanks.
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks><feature-extraction><filters><weights-initialization><feature-maps>
Title: Why CNN filters (kernels) are randomly initialized?
UserDisplayName: COTHE
ThreadText: # Why CNN filters (kernels) are randomly initialized?
  
  (Asked by: username_0 on 2024-01-23)
  
  I learned that when CNN filters are defined, they are initialized with random weights and bias(Im not sure about bias).
  
  Then as learning step goes on, the weight values change and each filter makes its own feature map.
  
  What I don't understand is that, if filter is initialized with random values, is there any chance that
  
  1. different filters make the same feature map
  2. feature map varies every time it repeats.
  
  It seems little unefficient to initialize the weights of every filter randomly. More precisely, I think (in most of the networks) the number of filters is too small to get meaningful features.
  
  Is the second case is why CNN has randomness?
  
  **Tags:** convolutional-neural-networks, feature-extraction, filters, weights-initialization, feature-maps
  > **Comments:**
  > **username_1**: first of all its is best to initialize randomly according to some distribution, for instance if you are using Relu activation, initialize according He-normal, or He-uniform, see the corresponding papers for why. second, if you initialize all weights similarly well then they all be treated in the same way by the gradient descent, and converge to similar values. finally in stochastic optimization, you need some kind to diversity to be able to explore the search space and find the solution. You do not no where the optimum solution is you need to find it, diversity ensures that you have a chance.  
  > **username_2**: Can you please put your **specific question** in the title? Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-02-08. Score: 1 | Accepted answer
  
  It is usual to initialize parameters with "a good guess", when you have prior information, in order to help the model converging.
  
  However in deep learning most of the time you have no clue what the weights should be, so you initialize them randomly following certain schedules like **Xavier initialization** or **Kaiming**, that prevent gradients from vanishing or exploding (due to a bad initialization) by setting the mean activations to be zero and the variance of the activations to be constant accross all layers.
  
  Regarding your questions :
  
  1. Yes it is totally possible that multiples filters learn the same thing. In fact, a good part of the filters may be completly useless, which is the idea behing **network pruning** that tries to delete neurons in a network while keeping the same performances.
  2. Yes, the network you'll obtain after training will depend on the initialization. You can have different results depending on the initialization (just permuting the filters, for instance, give a new network with same capabilities), since anyway you will never get the global optimum for your task but only sub-optimal weights. However if trained properly, the final results should not very too much.
  
  
  ---
  
  