Answers.Body: <blockquote>
  <p>Does that mean you're running the same kernel ... but just multiple copies of that process in parallel?<br>
  Would that not result in it converging to the same weights and getting the same feature maps?</p>
  </blockquote>
  <p>I found a related question <a href="https://ai.stackexchange.com/questions/5092/how-are-the-kernels-initialized-in-a-convolutional-neural-network">here</a> about how filters are initialized.
  It looks like filters are randomly initialized. <a href="https://www.telesens.co/2018/04/09/initializing-weights-for-the-convolutional-and-fully-connected-layers/" rel="nofollow noreferrer">This</a> page builds a CNN from scratch, and there is this paper <em><a href="https://arxiv.org/pdf/1511.06422.pdf" rel="nofollow noreferrer">All you need is a good init, Mishkin 2016</a></em> that discusses exactly this issue.</p>
  <blockquote>
  <p>Layer-sequential unit-variance (LSUV) initialization – a simple method for weight
  initialization for deep net learning – is proposed. The method consists of the two
  steps. First, pre-initialize weights of each convolution or inner-product layer with
  orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one.</p>
  </blockquote>
  <p>So it looks like filters are initialized in a number of ways. In tensorflow's <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D#args_1" rel="nofollow noreferrer">Conv2D</a> it specifies:</p>
  <blockquote>
  <p><strong>kernel_initializer</strong> | Initializer for the kernel weights matrix (see <a href="https://www.tensorflow.org/api_docs/python/tf/keras/initializers" rel="nofollow noreferrer"><strong>keras.initializers</strong></a>).<br>Defaults to 'glorot_uniform'.</p>
  </blockquote>
  <p>So you are able to specify how you would like this to happen. I found <a href="https://www.researchgate.net/post/Does_multiple_CNN_Kernels_can_learn_the_same_feature" rel="nofollow noreferrer">this</a> discussion about filter convergence, and it seems to be an open research question. To that point I did find this research paper that address exactly this issue <a href="https://people.cs.umass.edu/%7Earunirc/downloads/pubs/redundant_filter_dltp2017.pdf" rel="nofollow noreferrer">Reducing Duplicate Filters in DNN, RoyChowdhury 2017</a></p>
  <blockquote>
  <p><strong>5.Conclusion</strong><br>
  We have shown that duplication of filters occurs more in
  MLPs than CNNs, and this appears to be an outcome of
  Reducing duplicate filters in neural networks
  over-parameterization in the fully-connected MLP model.
  Increasing the number of filters at a layer results in more
  duplicates for MLP and is less marked for CNNs. A method
  to reduce a network based on near-duplicate filters is introduced and is shown to work well for fully-connected nets
  in the regime of moderate compression. For CNNs, a norm-based pruning strategy works as well or better, underlining
  their differences with MLPs in how parameter redundancy
  is manifested</p>
  </blockquote>
  <p>So filters <em>can</em> converge, the problem gets more likely with more filters, but the combination of filters having different initializations and the nature of CNNs seem to mitigate this issue.</p>
  <p>I hope this helps to clarify the issue :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-03T06:35:33.087Z
Answers.Text: Hi, thanks for the answer. The question asks about *numbers* of filters, while this answer seems to talk about *initializing* them; I might be misunderstanding something here, but I don't quite see how this adresses the question asked. Could you please [edit] to clarify how this explains the *number* of filters? Thank you!
Answers.UserDisplayName: Mithical
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-03T06:42:43.137Z
Answers.Text: I was addressing this aspect of the OPs question: 
  
  
  ***Does that mean you're running the same kernel with the same stride at the same channels in the same directions but just multiple copies of that process in parallel***
  
  
  By showing that the kernels are all initialized with random values they are all distinct and not copies. Model training then refines them to useful values.
Answers.UserDisplayName: J.Kent
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-03T08:04:48.197Z
Answers.Text: @delta2echo Indeed you're correct in that, due to the random initialisation of weights, different filters can learn different things. Or actually it's not even *random initialisation* per se that's the key, just *different initialisation* is they key (where randomness is then the most common way to achieve that difference). It would be good to edit that final step into your answer though, it's currently left implicit. You already write that there is random initialisation, but not yet that this is what allows different filters to learn different things.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-03T05:12:05.727Z
Answers.LastActivityDate: 2022-06-03T17:50:45.537Z
Answers.LastEditDate: 2022-06-03T17:50:45.537Z
Answers.UserDisplayName: J.Kent
Body: <p>I understand how depth, kernel size, stride, and padding works when dealing with filters in a spatial convolution layer.</p>
  <p>What I don't understand is &quot;the number of filters&quot;. Does that mean you're running the same kernel with the same stride at the same channels in the same directions but just multiple copies of that process in parallel?</p>
  <p>Would that not result in it converging to the same weights and getting the same feature maps?</p>
  
Tags: <convolutional-neural-networks><filters>
Title: CNNs: What does "number of filters" mean?
UserDisplayName: Darren Rahnemoon
ThreadText: # CNNs: What does "number of filters" mean?
  
  (Asked by: username_0 on 2022-06-03)
  
  I understand how depth, kernel size, stride, and padding works when dealing with filters in a spatial convolution layer.
  
  What I don't understand is "the number of filters". Does that mean you're running the same kernel with the same stride at the same channels in the same directions but just multiple copies of that process in parallel?
  
  Would that not result in it converging to the same weights and getting the same feature maps?
  
  **Tags:** convolutional-neural-networks, filters
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-03. Score: 1
  
  > Does that mean you're running the same kernel ... but just multiple copies of that process in parallel?  
  > Would that not result in it converging to the same weights and getting the same feature maps?
  
  I found a related question [here](https://ai.stackexchange.com/questions/5092/how-are-the-kernels-initialized-in-a-convolutional-neural-network) about how filters are initialized.
  It looks like filters are randomly initialized. [This](https://www.telesens.co/2018/04/09/initializing-weights-for-the-convolutional-and-fully-connected-layers/) page builds a CNN from scratch, and there is this paper *[All you need is a good init, Mishkin 2016](https://arxiv.org/pdf/1511.06422.pdf)* that discusses exactly this issue.
  
  > Layer-sequential unit-variance (LSUV) initialization – a simple method for weight
  > initialization for deep net learning – is proposed. The method consists of the two
  > steps. First, pre-initialize weights of each convolution or inner-product layer with
  > orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one.
  
  So it looks like filters are initialized in a number of ways. In tensorflow's [Conv2D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D#args_1) it specifies:
  
  > **kernel\_initializer** | Initializer for the kernel weights matrix (see [**keras.initializers**](https://www.tensorflow.org/api_docs/python/tf/keras/initializers)).  
  > Defaults to 'glorot\_uniform'.
  
  So you are able to specify how you would like this to happen. I found [this](https://www.researchgate.net/post/Does_multiple_CNN_Kernels_can_learn_the_same_feature) discussion about filter convergence, and it seems to be an open research question. To that point I did find this research paper that address exactly this issue [Reducing Duplicate Filters in DNN, RoyChowdhury 2017](https://people.cs.umass.edu/%7Earunirc/downloads/pubs/redundant_filter_dltp2017.pdf)
  
  > **5.Conclusion**  
  > We have shown that duplication of filters occurs more in
  > MLPs than CNNs, and this appears to be an outcome of
  > Reducing duplicate filters in neural networks
  > over-parameterization in the fully-connected MLP model.
  > Increasing the number of filters at a layer results in more
  > duplicates for MLP and is less marked for CNNs. A method
  > to reduce a network based on near-duplicate filters is introduced and is shown to work well for fully-connected nets
  > in the regime of moderate compression. For CNNs, a norm-based pruning strategy works as well or better, underlining
  > their differences with MLPs in how parameter redundancy
  > is manifested
  
  So filters *can* converge, the problem gets more likely with more filters, but the combination of filters having different initializations and the nature of CNNs seem to mitigate this issue.
  
  I hope this helps to clarify the issue :)
  
  > **Comments:**
  > **username_2**: Hi, thanks for the answer. The question asks about *numbers* of filters, while this answer seems to talk about *initializing* them; I might be misunderstanding something here, but I don't quite see how this adresses the question asked. Could you please [edit] to clarify how this explains the *number* of filters? Thank you!  
  > **username_1**: I was addressing this aspect of the OPs question:    ***Does that mean you're running the same kernel with the same stride at the same channels in the same directions but just multiple copies of that process in parallel***   By showing that the kernels are all initialized with random values they are all distinct and not copies. Model training then refines them to useful values.  
  > **username_3**: @delta2echo Indeed you're correct in that, due to the random initialisation of weights, different filters can learn different things. Or actually it's not even *random initialisation* per se that's the key, just *different initialisation* is they key (where randomness is then the most common way to achieve that difference). It would be good to edit that final step into your answer though, it's currently left implicit. You already write that there is random initialisation, but not yet that this is what allows different filters to learn different things.  
  
  ---
  
  