Answers.Body: <p>No, nothing really prevents the weights from being different. In practice though they end up almost always different because it makes the model more expressive (i.e. more powerful), so gradient descent learns to do that. If a model has <span class="math-container">$n$</span> features, but 2 of them are the same, then the model effectively has <span class="math-container">$n-1$</span> features, which is a less expressive model than that of <span class="math-container">$n$</span> features, and therefore usually has a larger loss function.</p>
  <p>But even if the weights are different, some of them can be very similar. If you visualize the first layer of your convolution filters, and you have a large number of them (e.g. 100), you will see some of them are learning to detect roughly the same edges (same orientation and placement). These features are so similar, they are effectively redundant in the model and do not add to its predictive power.</p>
  <p>There's actually an entire field of research on identifying redundant features and pruning them. Le Cun shows in <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf" rel="noreferrer">Optimal Brain Damage</a> that pruning out redundant features not only make the model smaller and faster for inference, but can also even improve the model's accuracy.</p>
  <p><a href="https://blog.dataiku.com/making-neural-networks-smaller-for-better-deployment-solving-the-size-problem-of-cnns-using-network-pruning-with-keras" rel="noreferrer">Here</a> is a blog post for a high level overview of one of the pruning methods for more info.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-11T06:16:43.117Z
Answers.LastActivityDate: 2020-12-11T06:16:43.117Z
Answers.UserDisplayName: user3667125
Body: <p>I trained a simple model to recognize handwritten numbers from the mnist dataset. Here it is:</p>
  <pre><code>model = Sequential([
      Conv2D(filters=1, kernel_size=(3,1), padding='valid', strides=1, input_shape=(28, 28, 1)),
      Flatten(),
      Dense(10, activation='softmax')])
  </code></pre>
  <p>I experimented with varying the number of filters for the convolutional layer, while keeping other parameters constant(learning rate=0.0001, number of episodes=2000, training batch size=512). I used 1, 2, 4, 8, and 16 filters, and the model accuracy was 92-93% for each of them.</p>
  <p>From my understanding, during the training the filters may learn to recognize various types of edges in the image (e.g, vertical, horizontal, round). This experiment made me wonder whether any of the filters end up being duplicate -- having the same or similar weights. Is there anything that prevents them from that?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-30T16:18:39.230Z
Comments.Text: Since you've edited the title of this, it's changed into being the same as [my follow up question](https://ai.stackexchange.com/questions/25464/is-there-anything-that-ensures-that-convolutional-filters-end-up-different-from). Not sure what admins will do with it, but keep an eye on it in case. Thanks for the great question!
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-30T16:26:01.903Z
Comments.Text: Hey, I did just change the title -- I just revisited the question and I realized I've made a mistake by not including the "don't". This might have been confusing, since in the description I ask "whether any of the filters end up being duplicate"
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-30T16:32:29.073Z
Comments.Text: Yeah makes sense. Let's see if we can get an interesting response between our questions then.
Comments.UserDisplayName: Alexander Soare
Tags: <neural-networks><convolutional-neural-networks><convolution><filters>
Title: Is there anything that ensures that convolutional filters don't end up the same?
UserDisplayName: mark mark
ThreadText: # Is there anything that ensures that convolutional filters don't end up the same?
  
  (Asked by: username_0 on 2020-12-11)
  
  I trained a simple model to recognize handwritten numbers from the mnist dataset. Here it is:
  
  ```
  model = Sequential([
      Conv2D(filters=1, kernel_size=(3,1), padding='valid', strides=1, input_shape=(28, 28, 1)),
      Flatten(),
      Dense(10, activation='softmax')])
  
  ```
  
  I experimented with varying the number of filters for the convolutional layer, while keeping other parameters constant(learning rate=0.0001, number of episodes=2000, training batch size=512). I used 1, 2, 4, 8, and 16 filters, and the model accuracy was 92-93% for each of them.
  
  From my understanding, during the training the filters may learn to recognize various types of edges in the image (e.g, vertical, horizontal, round). This experiment made me wonder whether any of the filters end up being duplicate -- having the same or similar weights. Is there anything that prevents them from that?
  
  **Tags:** neural-networks, convolutional-neural-networks, convolution, filters
  
  > **Comments:**
  > **username_1**: Since you've edited the title of this, it's changed into being the same as [my follow up question](https://ai.stackexchange.com/questions/25464/is-there-anything-that-ensures-that-convolutional-filters-end-up-different-from). Not sure what admins will do with it, but keep an eye on it in case. Thanks for the great question!  
  > **username_0**: Hey, I did just change the title -- I just revisited the question and I realized I've made a mistake by not including the "don't". This might have been confusing, since in the description I ask "whether any of the filters end up being duplicate"  
  > **username_1**: Yeah makes sense. Let's see if we can get an interesting response between our questions then.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-11. Score: 6
  
  No, nothing really prevents the weights from being different. In practice though they end up almost always different because it makes the model more expressive (i.e. more powerful), so gradient descent learns to do that. If a model has $n$ features, but 2 of them are the same, then the model effectively has $n-1$ features, which is a less expressive model than that of $n$ features, and therefore usually has a larger loss function.
  
  But even if the weights are different, some of them can be very similar. If you visualize the first layer of your convolution filters, and you have a large number of them (e.g. 100), you will see some of them are learning to detect roughly the same edges (same orientation and placement). These features are so similar, they are effectively redundant in the model and do not add to its predictive power.
  
  There's actually an entire field of research on identifying redundant features and pruning them. Le Cun shows in [Optimal Brain Damage](http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf) that pruning out redundant features not only make the model smaller and faster for inference, but can also even improve the model's accuracy.
  
  [Here](https://blog.dataiku.com/making-neural-networks-smaller-for-better-deployment-solving-the-size-problem-of-cnns-using-network-pruning-with-keras) is a blog post for a high level overview of one of the pruning methods for more info.
  
  
  ---
  
  