Answers.Body: <p>Yes, your thought experiment is correct, and the concept is known as <strong>breaking the symmetry</strong>. This is why biases can be initialized to <span class="math-container">$0$</span> (bias initialization doesn't matter), but weights should be randomly initialized to different numbers -- to break the symmetry. Otherwise, if not, the network will function as if it has <span class="math-container">$n-1$</span> filters (or however many filters that are unique) instead of the full <span class="math-container">$n$</span> filters.</p>
  <p>As for your main question, if two filters are initialized to very similar values, they may branch out as long as that is what minimizes the training loss. There is no collusion or coordination going on; each filter updates completely independently. You can even freeze all the other filters and only perform gradient descent on one filter at a time. Each filter just follows the direction of their gradient to minimize the training loss.</p>
  <p>Consider the backprop equations as defined by <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">this</a> online book:</p>
  <p><a href="https://i.sstatic.net/D5xG0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/D5xG0.png" alt="enter image description here" /></a></p>
  <p>The gradient of the current layer's weights depends on</p>
  <ol>
  <li>The future layers' weights, errors, and activation function's derivatives</li>
  <li>The current layer's activation function's derivative, and</li>
  <li>The previous layer's outputs.</li>
  </ol>
  <p>Each weight in the layer (i.e. each filter in the layer) looks at different parts of these three components (indexed by <span class="math-container">$j$</span> and <span class="math-container">$k$</span> in equation <span class="math-container">$BP4$</span>). It is this different perspective that allows them to update their gradients in different directions, even if their initial weights are very similar to each other. Note that it is possible that they end up with the same gradient, but it is very unlikely.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T09:23:58.927Z
Answers.Text: Thanks. I'm glad to have my initial thought experiment sense checked. As for the second part, I'm aware of the equations and the mechanisms of back propagation. And it sounds like you conclude with something like: "nothing guarantees that the filters diverge. it's possible that they randomly converge". Cool, this may be the answer. It's weird for me coming from a physics background though - because in mathematics and nature systems similar to this (very hand wavy I know) tend to either converge or diverge.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T09:25:26.897Z
Answers.Text: Even though the equations and mechanisms don't express explicit interaction between filters, I can't help but think that the system as a whole might implicitly do it. I'll leave this question open for a little while to see if anyone else has something to say.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T07:56:30.857Z
Answers.LastActivityDate: 2020-12-31T08:40:11.153Z
Answers.LastEditDate: 2020-12-31T08:40:11.153Z
Answers.UserDisplayName: user3667125
Answers.Body: <p>Here I am just trying to simplify what @user3667125 already said uses math arguments</p>
  <p>Say we have a cost function <span class="math-container">$J(x, y; F(\cdot; \Theta))$</span> which regards training a NN <span class="math-container">$F(\cdot, \Theta)$</span> with <span class="math-container">$x$</span> input and <span class="math-container">$y$</span> expected output</p>
  <p>Gradient descent tells us how to upgrade each <span class="math-container">$\theta_{i} \in \Theta$</span> and it is</p>
  <p><span class="math-container">$$ \theta_{i}(t+1) = \theta_{i}(t) - \alpha \frac{\partial J(x,y; \Theta)}{\partial \theta_{i}} $$</span></p>
  <p>with <span class="math-container">$t$</span> training time</p>
  <p>So let's focus on a specific component of the NN <span class="math-container">$f(x; \theta)$</span> and from its perspective, we can say the computation is</p>
  <p><span class="math-container">$$ h(f(g(x); \theta), y) $$</span></p>
  <p>with</p>
  <ul>
  <li><span class="math-container">$h(x,y)$</span> representing all the subsequent components + loss function</li>
  <li><span class="math-container">$g(x)$</span> representing all the previous computation</li>
  <li><span class="math-container">$x$</span> input</li>
  <li><span class="math-container">$y$</span> expected output</li>
  </ul>
  <p>so its update is</p>
  <p><span class="math-container">$$ \Delta \theta(t) = \frac{\partial h(f(g(x(t), \theta), y(t)))}{\partial \theta} $$</span></p>
  <p>with</p>
  <ul>
  <li><span class="math-container">$x(t)$</span> the concrete input at <span class="math-container">$t$</span> time</li>
  <li><span class="math-container">$y(t)$</span> expected output at <span class="math-container">$t$</span> time</li>
  <li><span class="math-container">$\theta(t)$</span> the concrete value of the parameter at <span class="math-container">$t$</span> time</li>
  </ul>
  <p>Applying the chain rule we have</p>
  <p><span class="math-container">$$ \Delta \theta(t) = h'(f(g(x(t), \theta(t))), y(t)) f'(g(x(t)), \theta(t)) $$</span></p>
  <p>so the gradient observed by a certain parameter <span class="math-container">$\theta$</span> depends on</p>
  <ul>
  <li><span class="math-container">$\theta(t)$</span> the current value of the parameter</li>
  <li><span class="math-container">$x(t)$</span> the current input and more specifically by <span class="math-container">$g(x(t))$</span> the processing of this input by the previous part of the NN</li>
  <li><span class="math-container">$y(t)$</span> the expected output</li>
  <li><span class="math-container">$h'(\cdot)$</span> the gradient of the subsequent part of the network</li>
  </ul>
  <p>So even if we have 2 weights with the same value <span class="math-container">$\theta_{i}(t) = \theta_{j}(t) \quad i \neq j$</span> at a certain point in training time, they can still see different gradients since</p>
  <ul>
  <li>the upstream processing <span class="math-container">$g(x(t))$</span> can be different</li>
  <li>the gradient backpropagating from the downstream processing <span class="math-container">$h'(\cdot, y(t))$</span> can be different</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T19:09:25.837Z
Answers.Text: Thanks for the breakdown. I think that two filters in the same layer will have the same downstream and upstream values and gradients. Therefore if we have two filters with all same values they will evolve in the same way.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T17:45:21.243Z
Answers.LastActivityDate: 2021-01-01T07:30:58.970Z
Answers.LastEditDate: 2021-01-01T07:30:58.970Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>I found this <a href="https://ai.stackexchange.com/questions/25109/is-there-anything-that-ensures-that-convolutional-filters-dont-end-up-the-same">question</a> very interesting, and this is a follow up on it.</p>
  <p>Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as possible (in terms of extracting useful information from the previous layer), without overlapping with another filter.</p>
  <p>A quick thought experiment tells me (please correct me if I'm wrong) that if two filters are identical down to maximum precision, then without adding in any other form of stochastic differentiation between them, their weights will be updated in the same way at each step of gradient descent during training. Thus, it would be a very bad idea to initialise all filters in the same way prior to training, as they would all be updated in exactly the same way (see footnote 1).</p>
  <p>On the other hand, a quick thought experiment isn't enough to tell me what would happen to two filters that are <em>almost</em> identical, as we continue to train the network. <strong>Is there some mechanism causing them to then diverge away from one another, thereby filling their own &quot;complementary niches&quot; in the layer?</strong> My intuition tells me that there must be, otherwise using many filters just wouldn't work. But during back-propagation, each filter is downstream, and so they don't have any way of communicating with one another. At the risk of anthropomorphising the network, I might ask &quot;How do the two filters collude with one another to benefit the network as a whole?&quot;</p>
  <hr />
  <p>Footnotes:</p>
  <ol>
  <li>Why do I think this? Because the expession for the partial derivative of the <span class="math-container">$k$</span>th filter weights with respect to the cost <span class="math-container">$\partial W^k/\partial C$</span> will be identical for all <span class="math-container">$k$</span>. From the perspective of back-propagation, all paths through the filters look exactly the same.</li>
  </ol>
  
Tags: <convolutional-neural-networks><gradient-descent>
Title: Is there anything that ensures that convolutional filters end up different from one another?
UserDisplayName: Alexander Soare
ThreadText: # Is there anything that ensures that convolutional filters end up different from one another?
  
  (Asked by: username_0 on 2020-12-30)
  
  I found this [question](https://ai.stackexchange.com/questions/25109/is-there-anything-that-ensures-that-convolutional-filters-dont-end-up-the-same) very interesting, and this is a follow up on it.
  
  Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as possible (in terms of extracting useful information from the previous layer), without overlapping with another filter.
  
  A quick thought experiment tells me (please correct me if I'm wrong) that if two filters are identical down to maximum precision, then without adding in any other form of stochastic differentiation between them, their weights will be updated in the same way at each step of gradient descent during training. Thus, it would be a very bad idea to initialise all filters in the same way prior to training, as they would all be updated in exactly the same way (see footnote 1).
  
  On the other hand, a quick thought experiment isn't enough to tell me what would happen to two filters that are *almost* identical, as we continue to train the network. **Is there some mechanism causing them to then diverge away from one another, thereby filling their own "complementary niches" in the layer?** My intuition tells me that there must be, otherwise using many filters just wouldn't work. But during back-propagation, each filter is downstream, and so they don't have any way of communicating with one another. At the risk of anthropomorphising the network, I might ask "How do the two filters collude with one another to benefit the network as a whole?"
  
  ---
  
  Footnotes:
  
  1. Why do I think this? Because the expession for the partial derivative of the $k$th filter weights with respect to the cost $\partial W^k/\partial C$ will be identical for all $k$. From the perspective of back-propagation, all paths through the filters look exactly the same.
  
  **Tags:** convolutional-neural-networks, gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-12-31. Score: 1
  
  Yes, your thought experiment is correct, and the concept is known as **breaking the symmetry**. This is why biases can be initialized to $0$ (bias initialization doesn't matter), but weights should be randomly initialized to different numbers -- to break the symmetry. Otherwise, if not, the network will function as if it has $n-1$ filters (or however many filters that are unique) instead of the full $n$ filters.
  
  As for your main question, if two filters are initialized to very similar values, they may branch out as long as that is what minimizes the training loss. There is no collusion or coordination going on; each filter updates completely independently. You can even freeze all the other filters and only perform gradient descent on one filter at a time. Each filter just follows the direction of their gradient to minimize the training loss.
  
  Consider the backprop equations as defined by [this](http://neuralnetworksanddeeplearning.com/chap2.html) online book:
  
  [![enter image description here](https://i.sstatic.net/D5xG0.png)](https://i.sstatic.net/D5xG0.png)
  
  The gradient of the current layer's weights depends on
  
  1. The future layers' weights, errors, and activation function's derivatives
  2. The current layer's activation function's derivative, and
  3. The previous layer's outputs.
  
  Each weight in the layer (i.e. each filter in the layer) looks at different parts of these three components (indexed by $j$ and $k$ in equation $BP4$). It is this different perspective that allows them to update their gradients in different directions, even if their initial weights are very similar to each other. Note that it is possible that they end up with the same gradient, but it is very unlikely.
  
  > **Comments:**
  > **username_0**: Thanks. I'm glad to have my initial thought experiment sense checked. As for the second part, I'm aware of the equations and the mechanisms of back propagation. And it sounds like you conclude with something like: "nothing guarantees that the filters diverge. it's possible that they randomly converge". Cool, this may be the answer. It's weird for me coming from a physics background though - because in mathematics and nature systems similar to this (very hand wavy I know) tend to either converge or diverge.  
  > **username_0**: Even though the equations and mechanisms don't express explicit interaction between filters, I can't help but think that the system as a whole might implicitly do it. I'll leave this question open for a little while to see if anyone else has something to say.  
  
  ---
  
  ### Answer by username_2 on 2020-12-31. Score: 0
  
  Here I am just trying to simplify what @user3667125 already said uses math arguments
  
  Say we have a cost function $J(x, y; F(\cdot; \Theta))$ which regards training a NN $F(\cdot, \Theta)$ with $x$ input and $y$ expected output
  
  Gradient descent tells us how to upgrade each $\theta\_{i} \in \Theta$ and it is
  
  $$ \theta\_{i}(t+1) = \theta\_{i}(t) - \alpha \frac{\partial J(x,y; \Theta)}{\partial \theta\_{i}} $$
  
  with $t$ training time
  
  So let's focus on a specific component of the NN $f(x; \theta)$ and from its perspective, we can say the computation is
  
  $$ h(f(g(x); \theta), y) $$
  
  with
  
  * $h(x,y)$ representing all the subsequent components + loss function
  * $g(x)$ representing all the previous computation
  * $x$ input
  * $y$ expected output
  
  so its update is
  
  $$ \Delta \theta(t) = \frac{\partial h(f(g(x(t), \theta), y(t)))}{\partial \theta} $$
  
  with
  
  * $x(t)$ the concrete input at $t$ time
  * $y(t)$ expected output at $t$ time
  * $\theta(t)$ the concrete value of the parameter at $t$ time
  
  Applying the chain rule we have
  
  $$ \Delta \theta(t) = h'(f(g(x(t), \theta(t))), y(t)) f'(g(x(t)), \theta(t)) $$
  
  so the gradient observed by a certain parameter $\theta$ depends on
  
  * $\theta(t)$ the current value of the parameter
  * $x(t)$ the current input and more specifically by $g(x(t))$ the processing of this input by the previous part of the NN
  * $y(t)$ the expected output
  * $h'(\cdot)$ the gradient of the subsequent part of the network
  
  So even if we have 2 weights with the same value $\theta\_{i}(t) = \theta\_{j}(t) \quad i \neq j$ at a certain point in training time, they can still see different gradients since
  
  * the upstream processing $g(x(t))$ can be different
  * the gradient backpropagating from the downstream processing $h'(\cdot, y(t))$ can be different
  
  > **Comments:**
  > **username_0**: Thanks for the breakdown. I think that two filters in the same layer will have the same downstream and upstream values and gradients. Therefore if we have two filters with all same values they will evolve in the same way.  
  
  ---
  
  