Answers.Body: <p>The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the <strong>3D filter</strong> (grey cube) is made up of a different value (<code>3x3x3=27</code> values). So, three different <strong>2D filters</strong> of size <code>3x3</code> can be concatenated to form this one <strong>3D filter</strong> of size <code>3x3x3</code>.</p>
  <p><a href="https://i.sstatic.net/YCJRm.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YCJRm.jpg" alt="convnet2D" /></a></p>
  <p>The <code>3x3x3</code> RGB chunk from the picture is multiplied <em>elementwise</em> by a <strong>3D filter</strong> (shown as grey). In this case, the filter has <code>3x3x3=27</code> weights. When these weights are multiplied element-wise and then summed, it gives one value.</p>
  <h3>So, is there a separate filter for each input channel?</h3>
  <p><strong>YES</strong>, there are as many <strong>2D filters</strong> as the number of input channels in the image. <strong>However</strong>, it helps if you think that for input matrices with more than one channel, there is only one <strong>3D filter</strong> (as shown in the image above).</p>
  <h3>Then why is this called 2D convolution (if the filter is 3D and the input matrix is 3D)?</h3>
  <p>This is 2D convolution because the strides of the filter are along the height and width dimensions only (<strong>NOT</strong> depth) and therefore, the output produced by this convolution is also a 2D matrix. The number of movement directions of the filter determines the dimensions of convolution.</p>
  <p><strong>Note:</strong> <em>If you build up your understanding by visualizing a single <strong>3D filter</strong> instead of multiple <strong>2D filters</strong> (one for each layer), then you will have an easy time understanding advanced CNN architectures like Resnet, InceptionV3, etc.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-23T10:48:16.010Z
Answers.Text: @pentanol see the original CNN paper and understand the mathematics. Until then the answer above is the resource itself. Good Luck!
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-12T15:00:02.827Z
Answers.Text: @MohsinBukhari Is there any resources illustrating what you say ?
Answers.UserDisplayName: pentanol
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-22T23:25:44.220Z
Answers.Text: this is a good explanation, but more specifically the question I'm trying to understand is whether the filters that operate on each input channel are copies of the same weights, or completely different weights.  This isn't actually shown in the image and in fact to me that image kind of suggests that it's the same weights applied to each channel (since their the same color)... Per @neil slater 's answer, it sounds like each filter actually has number of `input_channels` versions with __different__ weights.  If this is also your understanding, is there an "official" source  confirming this?
Answers.UserDisplayName: Ryan Chase
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-23T10:59:44.657Z
Answers.Text: Yes, indeed, that's also my understanding. For me, that was clear when I tried to think of that **grey** cube to be composed of 27 different weight values. This means that there are 3 different 2D filters rather the same 2D filter applied to each input layer.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-23T11:01:17.133Z
Answers.Text: I could not find any official source for confirming this. However, when I was trying to wrap my head around this same concept, I created a dummy input and weight filter in Tensorflow and observed the output. I was content with that. If I find any **official** explanation. I will edit my answer above.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-23T11:03:00.913Z
Answers.Text: If you follow the Tensorflow path. You can print your weight filter after showing your dummy CNN layer an input sample.
Answers.UserDisplayName: Mohsin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-25T00:58:20.790Z
Answers.Text: @Moshsin Bukhari I will definitely try to explore the filters within TensorFlow.  Would you be willing to share your code for how you went about about exploring what's contained in the filters?  Are you able to print the values of the filter at each step in the network for example?
Answers.UserDisplayName: Ryan Chase
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-22T19:41:57.567Z
Answers.LastActivityDate: 2021-12-18T22:46:32.797Z
Answers.LastEditDate: 2021-12-18T22:46:32.797Z
Answers.UserDisplayName: Mohsin
Answers.Body: <blockquote>
    <p>In a convolutional neural network, is there a <em>unique</em> filter for each input channel or are the same new filters used across all input channels?</p>
  </blockquote>
  
  <p>The former. In fact there is a <em>separate</em> kernel defined for each input channel / output channel combination. </p>
  
  <p>Typically for a CNN architecture, in a single filter as described by your <code>number_of_filters</code> parameter, there is one 2D kernel per input channel. There are <code>input_channels * number_of_filters</code> sets of weights, each of which describe a convolution kernel. So the diagrams showing one set of weights per input channel for each filter are correct. The first diagram also shows clearly that the results of applying those kernels are combined by summing them up and adding bias for each output channel.</p>
  
  <p>This can <em>also</em> be viewed as using a 3D convolution for each output channel, that happens to have the same depth as the input. Which is what your second diagram is showing, and also what many libraries will do internally. Mathematically this is the same result (provided the depths match exactly), although the layer type is typically labelled as "Conv2D" or similar. Similarly if your input type is inherently 3D, such as voxels or a video, then you might use a "Conv3D" layer, but internally it could well be implemented as a 4D convolution.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-22T23:28:50.120Z
Answers.Text: thanks for this explanation.  It sounds like each filter actually has number of `input_channels` versions with __different weights__.  Do you have an "official" source that confirms this understanding?
Answers.UserDisplayName: Ryan Chase
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-23T10:26:05.630Z
Answers.Text: @RyanChase: Yes that is correct. I would just point you at Andrew Ng's course on CNNs - starting here with how colour image would be processed:  https://www.coursera.org/learn/convolutional-neural-networks/lecture/ctQZz/convolutions-over-volume
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-13T22:32:35.807Z
Answers.Text: I  would like to note that, in that source (http://cs231n.github.io/convolutional-networks/), filters (weights or kernesl) are volumes (i.e. 3-dimensional), and they have the same 3rd dimension has the one of the input volume. Furthermore, as it is (at least) now stated in that source, the volumes have been sliced across the 3rd dimension in order to visualize better the application of the filter to the input volume. I don't think that, in general, "there is a separate kernel defined for each input channel / output channel combination." is correct.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-13T22:39:06.623Z
Answers.Text: Note that the filters (or kernels) are the weights that need to be learned (i.e. they are not fixed, but they are actually the parameters of the CNN). It might be that they are (i.e. the slices of the filter), at the end, the same across the 3rd dimension.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T07:46:09.403Z
Answers.Text: @nbro: Yes you can *implement* a 2D convolution across multiple 2D slices as a single 3D convolution with the kernel depth same as number of channels. Mathematically this is identical to my description. You can also view it as a truncated fully-connected feed forward network with shared weights (many of which are zero). This answer focuses on what the view of 2D filters is, because the OP is asking about how the 2D filters are arranged. They may in fact be arranged into a larger 3D kernel, but they are still applied as 2D kernels using the "trick" that the 3D convolution is equivalent.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-22T08:24:51.310Z
Answers.LastActivityDate: 2018-03-22T09:37:59.200Z
Answers.LastEditDate: 2018-03-22T09:37:59.200Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I recommend <a href="https://arxiv.org/pdf/1707.09725.pdf" rel="nofollow noreferrer">chapter 2.2.1 of my masters thesis</a> as an answer. To add to the remaining answers:</p>
  
  <p>Keras is your friend to understand what happens:</p>
  
  <pre><code>from keras.models import Sequential
  from keras.layers import Conv2D
  
  model = Sequential()
  model.add(Conv2D(32, input_shape=(28, 28, 3),
            kernel_size=(5, 5),
            padding='same',
            use_bias=False))
  model.add(Conv2D(17, (3, 3), padding='same', use_bias=False))
  model.add(Conv2D(13, (3, 3), padding='same', use_bias=False))
  model.add(Conv2D(7, (3, 3), padding='same', use_bias=False))
  model.compile(loss='categorical_crossentropy', optimizer='adam')
  
  print(model.summary())
  </code></pre>
  
  <p>gives</p>
  
  <pre><code>_________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  conv2d_1 (Conv2D)            (None, 28, 28, 32)        2400      
  _________________________________________________________________
  conv2d_2 (Conv2D)            (None, 28, 28, 17)        4896      
  _________________________________________________________________
  conv2d_3 (Conv2D)            (None, 28, 28, 13)        1989      
  _________________________________________________________________
  conv2d_4 (Conv2D)            (None, 28, 28, 7)         819       
  =================================================================
  Total params: 10,104
  </code></pre>
  
  <p>Try to formulate your options. What would that mean for the parameters if something else would be the case?</p>
  
  <p>Hint: <span class="math-container">$2400 = 32 \cdot (3 \cdot 5 \cdot 5)$</span></p>
  
  <p>This approach also helps you with other layer types, not only convolutional layers.</p>
  
  <p>Please also note that you are free to implement different solutions, that might have other numbers of parameters. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-26T14:43:53.190Z
Answers.LastActivityDate: 2018-12-04T09:51:44.970Z
Answers.LastEditDate: 2018-12-04T09:51:44.970Z
Answers.UserDisplayName: Martin Thoma
Answers.Body: <p>I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively:</p>
  
  <p>Let the example be as follows (wrt to 1 convolutional layer):</p>
  
  <ul>
  <li>the input tensor is 9x9x5, i.e. 5 input channels, so <code>input_channels=5</code></li>
  <li>the filter/kernel size is 4x4 and the stride is 1</li>
  <li>the output tensor is 6x6x56, i.e. 56 output channels, so <code>output_channels=56</code></li>
  <li>the padding type is 'VALID' (i.e. no padding)</li>
  </ul>
  
  <p>We note that:</p>
  
  <ul>
  <li>since the input has 5 channels, the filter dimension becomes 4x4x5, i.e. there are 5 separate, unique 2D filters of size 4x4 (i.e. each has 16 weights); in order to convolve over the input of size 9x9x5 the filter becomes 3D and must be of size 4x4x5</li>
  <li>therefore: for each input channel, there exists a distinct 2D filter with 16 different weights each. In other words, the number of 2D filters matches the number of input channels</li>
  <li>since there are 56 output channels, there must be 56 3-dimensional filters W0, W1, ..., W55 of size 4x4x5 (cf. in the CS231 graphic there are 2 3-dimensional filters W0, W1 to account for the 2 output channels), where the 3rd dimension of size 5 represents the link to the 5 input channels (cf. in the CS231 graphic each 3D filter W0, W1 has the 3rd dimension 3, which matches the 3 input channels)</li>
  <li>therefore: the number of 3D filters equals the number of output channels</li>
  </ul>
  
  <p>That convolutional layer thus contains: </p>
  
  <p>56 3-dimensional filters of size 4x4x5 (= 80 different weights each) to account for the 56 output channels where each has a value for the 3rd dimension of 5 to match the 5 input channels. In total there are </p>
  
  <p><code>number_of_filters=input_channel*output_channels=5*56=280</code> </p>
  
  <p>2D filters of size 4x4 (i.e. 280x16 different weights in total).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-25T16:46:23.593Z
Answers.LastActivityDate: 2018-11-26T08:19:42.970Z
Answers.LastEditDate: 2018-11-26T08:19:42.970Z
Answers.UserDisplayName: Lukas Z.
Answers.Body: <p>One addition to Mohsin's answer: try the awesome <a href="https://poloclub.github.io/cnn-explainer/" rel="nofollow noreferrer">CNN explainer</a>. There you can explore CNN's graphically, which makes things &quot;click&quot; very fast.</p>
  <p>If you click on conv_1_1, you will see how three different kernels are used to calculate one convolution. That means you have one 2D filter per channel, that could together be interpreted as one 3D filter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T17:21:57.560Z
Answers.Text: While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/23959)
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T14:05:22.057Z
Answers.LastActivityDate: 2023-01-10T14:05:22.057Z
Answers.UserDisplayName: Straighter
Answers.Body: <p>Other answers claim that you have a different <strong>2-dimensional</strong> kernel (i.e. a matrix) for each channel. This is not wrong, but it's just a <em>conceptual interpretation</em> of the 2d convolution, which emphasizes that different channels may provide different information during the convolution: this is the only advantage that I see of this interpretation!</p>
  <p>Another interpretation (which I think has more advantages, as I will explain below) is to think of a kernel as a 3d multi-dimensional array that has the same depth as the input. So, for example, if you have an input (an image or feature map) of shape <span class="math-container">$H \times W \times D$</span>, then a single kernel needs to have the shape <span class="math-container">$K \times L \times D$</span>, where <span class="math-container">$D$</span> is the depth of both the input and kernel. If you have more than one kernel, they will all have the depth <span class="math-container">$D$</span>. This interpretation is consistent with the excerpt that you are quoting.</p>
  <p>This second interpretation is conceptually useful when you need to deal with 3d convolutions: if you follow this interpretation, a 3d convolution is just a convolution where kernels don't necessarily have the same depth as the input (and that would be the only difference between 2d and 3d convolutions). The other advantage of this interpretation is that your confusion would not have arisen in the first place: your confusion arouse because you thought that kernels are 2d arrays. Another advantage is that, if you have <span class="math-container">$N &gt; 1$</span> kernels (which is the usual case), then you know immediately that <span class="math-container">$N$</span> is the depth of the output volume and you don't need to think about multiple channels of the input volume: you can simply think in terms of multi-dimensional arrays.</p>
  <p>So, to clarify, in case it wasn't already super clear: <strong>no, you don't apply the same 2d matrix to multiple channels of the input!</strong></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-18T23:13:16.193Z
Answers.LastActivityDate: 2021-12-18T23:13:16.193Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Just to make two details absolutely clear:</p>
  
  <p>Say you have <span class="math-container">$N$</span> 2D input channels going to <span class="math-container">$N$</span> 2D output channels.  The total number of 2D <span class="math-container">$3\times3$</span> filter weights is actually <span class="math-container">$N^2$</span>.  But how is the 3D convolution affected, i.e., if every input channel contributes one 2D layer to every output channel, then each output channel is composed initially of <span class="math-container">$N$</span> 2D layers, how are they combined?</p>
  
  <p>This tends to be glossed over in almost every publication I've seen, but the key concept is the <span class="math-container">$N^2$</span> 2D output channels are interleaved with each other to form the <span class="math-container">$N$</span> output channels, like shuffled card decks, before being summed together.  This is all logical when you realize that along the channel dimensions of a convolution (which is never illustrated), you actually have a fully connected layer!  Every input 2D channel, multiplied by a unique <span class="math-container">$3\times 3$</span> filter, yields a 2D output layer contribution to a single output channel.  Once combined, every output layer is a combination of every input layer <span class="math-container">$\times$</span> a unique filter.  It's an all to all contribution.</p>
  
  <p>The easiest way to convince yourself of this is to imagine what happens in other scenarios and see that the computation becomes degenerate - that is, if you don't interleave and recombine the results, then the different outputs wouldn't actually do anything - they'd have the same effect as a single output with combined weights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-26T15:25:34.630Z
Answers.LastActivityDate: 2019-03-03T17:57:20.323Z
Answers.LastEditDate: 2019-03-03T17:57:20.323Z
Answers.UserDisplayName: user2465201
Answers.Body: <p>For anyone trying to understand how convolutions are calculated, here is a useful code snippet in Pytorch:</p>
  
  <pre><code>batch_size = 1
  height = 3 
  width = 3
  conv1_in_channels = 2
  conv1_out_channels = 2
  conv2_out_channels = 2
  kernel_size = 2
  # (N, C_in, H, W) is shape of all tensors. (batch_size, channels, height, width)
  input = torch.Tensor(np.arange(0, batch_size*height*width*in_channels).reshape(batch_size, in_channels, height, width))
  conv1 = nn.Conv2d(in_channels, conv1_out_channels, kernel_size, bias=False) # no bias to make calculations easier
  # set the weights of the convolutions to make the convolutions easier to follow
  nn.init.constant_(conv1.weight[0][0], 0.25)
  nn.init.constant_(conv1.weight[0][1], 0.5)
  nn.init.constant_(conv1.weight[1][0], 1) 
  nn.init.constant_(conv1.weight[1][1], 2) 
  out1 = conv1(input) # compute the convolution
  
  conv2 = nn.Conv2d(conv1_out_channels, conv2_out_channels, kernel_size, bias=False)
  nn.init.constant_(conv2.weight[0][0], 0.25)
  nn.init.constant_(conv2.weight[0][1], 0.5)
  nn.init.constant_(conv2.weight[1][0], 1) 
  nn.init.constant_(conv2.weight[1][1], 2) 
  out2 = conv2(out1) # compute the convolution
  
  for tensor, name in zip([input, conv1.weight, out1, conv2.weight, out2], ['input', 'conv1', 'out1', 'conv2', 'out2']):
      print('{}: {}'.format(name, tensor))
      print('{} shape: {}'.format(name, tensor.shape))
  </code></pre>
  
  <p>Running this gives the following output:</p>
  
  <pre><code>input: tensor([[[[ 0.,  1.,  2.],
            [ 3.,  4.,  5.],
            [ 6.,  7.,  8.]],
  
           [[ 9., 10., 11.],
            [12., 13., 14.],
            [15., 16., 17.]]]])
  input shape: torch.Size([1, 2, 3, 3])
  conv1: Parameter containing:
  tensor([[[[0.2500, 0.2500],
            [0.2500, 0.2500]],
  
           [[0.5000, 0.5000],
            [0.5000, 0.5000]]],
  
  
          [[[1.0000, 1.0000],
            [1.0000, 1.0000]],
  
           [[2.0000, 2.0000],
            [2.0000, 2.0000]]]], requires_grad=True)
  conv1 shape: torch.Size([2, 2, 2, 2])
  out1: tensor([[[[ 24.,  27.],
            [ 33.,  36.]],
  
           [[ 96., 108.],
            [132., 144.]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
  out1 shape: torch.Size([1, 2, 2, 2])
  conv2: Parameter containing:
  tensor([[[[0.2500, 0.2500],
            [0.2500, 0.2500]],
  
           [[0.5000, 0.5000],
            [0.5000, 0.5000]]],
  
  
          [[[1.0000, 1.0000],
            [1.0000, 1.0000]],
  
           [[2.0000, 2.0000],
            [2.0000, 2.0000]]]], requires_grad=True)
  conv2 shape: torch.Size([2, 2, 2, 2])
  out2: tensor([[[[ 270.]],
  
           [[1080.]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
  out2 shape: torch.Size([1, 2, 1, 1])
  </code></pre>
  
  <p>Notice how the each channel of the convolution sums over all previous channels outputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T19:19:26.140Z
Answers.LastActivityDate: 2019-07-10T19:19:26.140Z
Answers.UserDisplayName: Simon Alford
Answers.Body: <p>My understanding from this paper <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/TASLP2339736-proof.pdf" rel="nofollow noreferrer">https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/TASLP2339736-proof.pdf</a> was that the filters are used on each input channels (i.e input feature map in the paper) separately and the result is summed, as described in eq. 8. Here they use different filters for each channel, but you could totally use the same filter.
  As to knowing the software implementation in a particular ML library such as Tensorflow or PyTorch, this requires inspection as suggested by @Mohsin Bukhari</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-22T15:09:37.653Z
Answers.LastActivityDate: 2021-11-22T15:09:37.653Z
Answers.UserDisplayName: Whynote
Answers.Body: <p>The Conv2d filter is defined as a 3D-represented tensor, but it's indeed a collection of num_input_channels kernels applied laterally. The different kernels' layers' values are distinct and learned distinctly too. Theoretically it's necessary for it to be that way; otherwise it wouldn't have the degrees of freedom to distinguish well across the input channels, e.g. find color features in the case of the first layer of a color image input!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-24T19:22:16.777Z
Answers.LastActivityDate: 2023-04-24T19:22:16.777Z
Answers.UserDisplayName: hafniate
Answers.Body: <p>Refer to &quot;Local Connectivity&quot; section in <a href="http://cs231n.github.io/convolutional-networks/" rel="nofollow noreferrer">here</a> and slide 7-18.</p>
  <p>&quot;Receptive Field&quot; hyperparameter of filter is defined by height &amp; width only, as depth is fixed by preceding layer's depth.</p>
  <p>NOTE that &quot;The extent of the connectivity along the depth axis is always equal to the DEPTH of the input volume&quot; -or- DEPTH of activation map (in case of later layers).</p>
  <p>Intuitively, this must be due to the fact that image channels data are interleaved, not planar. This way, applying filter can be achieved simply by column vectors multiplication.</p>
  <p>NOTE that Convolutional Network learns all the filter parameters (including the depth dimension) and they are total &quot;h<em>w</em>input_layer_depth + 1 (bias)&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-07T14:41:49.480Z
Answers.LastActivityDate: 2021-01-18T09:38:24.827Z
Answers.LastEditDate: 2021-01-18T09:38:24.827Z
Answers.UserDisplayName: sunil4data
Answers.Body: <p>There are only restriction in 2D. Why?</p>
  
  <p>Imagine a fully connected layer.</p>
  
  <p>It'd be awfully huge, each neuron would be connected to maybe 1000x1000x3 inputs neurons. But we know that processing nearby pixel makes sense, therefore we limit ourselves to a small 2D-neighborhood, so each neuron is connected to only a 3x3 near neurons in 2D. We don't know such a thing about channels, so we connect to all channels.</p>
  
  <p>Still, there would be too many weights. But because of the translation invariance, a filter working well in one area is most probably useful in a different area. So we use the same set of weights across 2D. Again, there's no such translation invariance between channels, so there's no such restriction there.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-23T04:39:52.307Z
Answers.LastActivityDate: 2018-03-23T04:39:52.307Z
Answers.UserDisplayName: maaartinus
Body: <p>My understanding is that the convolutional layer of a convolutional neural network has four dimensions: <code>input_channels, filter_height, filter_width, number_of_filters</code>.  Furthermore, it is my understanding that each new filter just gets convoluted over ALL of the <code>input_channels</code> (or feature/activation maps from the previous layer).</p>
  <p>HOWEVER, the graphic below from CS231 shows each filter (in red) being applied to a SINGLE CHANNEL, rather than the same filter being used across channels.  This seems to indicate that there is a separate filter for EACH channel (in this case I'm assuming they're the three color channels of an input image, but the same would apply for all input channels).</p>
  <p>This is confusing - is there a different unique filter for each input channel?</p>
  <p><a href="https://i.sstatic.net/3m7mW.png" rel="noreferrer"><img src="https://i.sstatic.net/3m7mW.png" alt="Convolutional filters diagram" /></a></p>
  <p>This is the <a href="http://cs231n.github.io/convolutional-networks/" rel="noreferrer">source</a>.</p>
  <p>The above image seems contradictory to an excerpt from O'reilly's <a href="https://rads.stackoverflow.com/amzn/click/com/1491925612" rel="noreferrer" rel="nofollow noreferrer">&quot;Fundamentals of Deep Learning&quot;</a>:</p>
  <blockquote>
  <p>...filters don't just operate on a single feature map. They operate on the entire volume of feature maps that have been generated at a particular layer...As a result, feature maps must be able to operate over volumes, not just areas</p>
  </blockquote>
  <p>...Also, it is my understanding that these images below are indicating a <strong>THE SAME</strong> filter is just convolved over all three input channels (contradictory to what's shown in the CS231 graphic above):</p>
  <p><a href="https://i.sstatic.net/VdqER.png" rel="noreferrer"><img src="https://i.sstatic.net/VdqER.png" alt="Application of a volumetric convolutional filter to an RGB image" /></a></p>
  <p><a href="https://i.sstatic.net/kczF0.png" rel="noreferrer"><img src="https://i.sstatic.net/kczF0.png" alt="Convolutions on an RGB image" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-02T07:17:31.303Z
Comments.Text: Great question and answers below. The thing that threw me was that you actually need the same amount of filters as the amount of output channels. So a 2dConv on an RGB image that outputs an RGB image will require 3 filters of shape [3,W,H]. So the actual volume of weights is [3,3,W,H]. I wrote an implementation in pytorch here. https://stackoverflow.com/a/62678843/5484902
Comments.UserDisplayName: Gal_M
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-26T12:12:26.517Z
Comments.Text: https://arxiv.org/pdf/1707.09725 chapter 2
Comments.UserDisplayName: Martin Thoma
Tags: <convolutional-neural-networks><weights><filters>
Title: In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
UserDisplayName: Ryan Chase
ThreadText: # In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
  
  (Asked by: username_0 on 2018-03-22)
  
  My understanding is that the convolutional layer of a convolutional neural network has four dimensions: `input_channels, filter_height, filter_width, number_of_filters`. Furthermore, it is my understanding that each new filter just gets convoluted over ALL of the `input_channels` (or feature/activation maps from the previous layer).
  
  HOWEVER, the graphic below from CS231 shows each filter (in red) being applied to a SINGLE CHANNEL, rather than the same filter being used across channels. This seems to indicate that there is a separate filter for EACH channel (in this case I'm assuming they're the three color channels of an input image, but the same would apply for all input channels).
  
  This is confusing - is there a different unique filter for each input channel?
  
  [![Convolutional filters diagram](https://i.sstatic.net/3m7mW.png)](https://i.sstatic.net/3m7mW.png)
  
  This is the [source](http://cs231n.github.io/convolutional-networks/).
  
  The above image seems contradictory to an excerpt from O'reilly's ["Fundamentals of Deep Learning"](https://rads.stackoverflow.com/amzn/click/com/1491925612):
  
  > ...filters don't just operate on a single feature map. They operate on the entire volume of feature maps that have been generated at a particular layer...As a result, feature maps must be able to operate over volumes, not just areas
  
  ...Also, it is my understanding that these images below are indicating a **THE SAME** filter is just convolved over all three input channels (contradictory to what's shown in the CS231 graphic above):
  
  [![Application of a volumetric convolutional filter to an RGB image](https://i.sstatic.net/VdqER.png)](https://i.sstatic.net/VdqER.png)
  
  [![Convolutions on an RGB image](https://i.sstatic.net/kczF0.png)](https://i.sstatic.net/kczF0.png)
  
  **Tags:** convolutional-neural-networks, weights, filters
  > **Comments:**
  > **username_2**: https://arxiv.org/pdf/1707.09725 chapter 2  
  > **username_1**: Great question and answers below. The thing that threw me was that you actually need the same amount of filters as the amount of output channels. So a 2dConv on an RGB image that outputs an RGB image will require 3 filters of shape [3,W,H]. So the actual volume of weights is [3,3,W,H]. I wrote an implementation in pytorch here. https://stackoverflow.com/a/62678843/5484902  
  
  ---
  
  ## 12 answers
  
  ### Answer by username_3 on 2018-03-22. Score: 27
  
  The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the **3D filter** (grey cube) is made up of a different value (`3x3x3=27` values). So, three different **2D filters** of size `3x3` can be concatenated to form this one **3D filter** of size `3x3x3`.
  
  [![convnet2D](https://i.sstatic.net/YCJRm.jpg)](https://i.sstatic.net/YCJRm.jpg)
  
  The `3x3x3` RGB chunk from the picture is multiplied *elementwise* by a **3D filter** (shown as grey). In this case, the filter has `3x3x3=27` weights. When these weights are multiplied element-wise and then summed, it gives one value.
  
  ### So, is there a separate filter for each input channel?
  
  **YES**, there are as many **2D filters** as the number of input channels in the image. **However**, it helps if you think that for input matrices with more than one channel, there is only one **3D filter** (as shown in the image above).
  
  ### Then why is this called 2D convolution (if the filter is 3D and the input matrix is 3D)?
  
  This is 2D convolution because the strides of the filter are along the height and width dimensions only (**NOT** depth) and therefore, the output produced by this convolution is also a 2D matrix. The number of movement directions of the filter determines the dimensions of convolution.
  
  **Note:** *If you build up your understanding by visualizing a single **3D filter** instead of multiple **2D filters** (one for each layer), then you will have an easy time understanding advanced CNN architectures like Resnet, InceptionV3, etc.*
  
  > **Comments:**
  > **username_0**: this is a good explanation, but more specifically the question I'm trying to understand is whether the filters that operate on each input channel are copies of the same weights, or completely different weights.  This isn't actually shown in the image and in fact to me that image kind of suggests that it's the same weights applied to each channel (since their the same color)... Per @neil slater 's answer, it sounds like each filter actually has number of `input_channels` versions with __different__ weights.  If this is also your understanding, is there an "official" source  confirming this?  
  > **username_3**: Yes, indeed, that's also my understanding. For me, that was clear when I tried to think of that **grey** cube to be composed of 27 different weight values. This means that there are 3 different 2D filters rather the same 2D filter applied to each input layer.  
  > **username_3**: I could not find any official source for confirming this. However, when I was trying to wrap my head around this same concept, I created a dummy input and weight filter in Tensorflow and observed the output. I was content with that. If I find any **official** explanation. I will edit my answer above.  
  > **username_3**: If you follow the Tensorflow path. You can print your weight filter after showing your dummy CNN layer an input sample.  
  > **username_0**: @Moshsin Bukhari I will definitely try to explore the filters within TensorFlow.  Would you be willing to share your code for how you went about about exploring what's contained in the filters?  Are you able to print the values of the filter at each step in the network for example?  
  > **username_4**: @MohsinBukhari Is there any resources illustrating what you say ?  
  > **username_3**: @pentanol see the original CNN paper and understand the mathematics. Until then the answer above is the resource itself. Good Luck!  
  
  ---
  
  ### Answer by username_5 on 2018-03-22. Score: 16
  
  > In a convolutional neural network, is there a *unique* filter for each input channel or are the same new filters used across all input channels?
  
  The former. In fact there is a *separate* kernel defined for each input channel / output channel combination.
  
  Typically for a CNN architecture, in a single filter as described by your `number_of_filters` parameter, there is one 2D kernel per input channel. There are `input_channels * number_of_filters` sets of weights, each of which describe a convolution kernel. So the diagrams showing one set of weights per input channel for each filter are correct. The first diagram also shows clearly that the results of applying those kernels are combined by summing them up and adding bias for each output channel.
  
  This can *also* be viewed as using a 3D convolution for each output channel, that happens to have the same depth as the input. Which is what your second diagram is showing, and also what many libraries will do internally. Mathematically this is the same result (provided the depths match exactly), although the layer type is typically labelled as "Conv2D" or similar. Similarly if your input type is inherently 3D, such as voxels or a video, then you might use a "Conv3D" layer, but internally it could well be implemented as a 4D convolution.
  
  > **Comments:**
  > **username_0**: thanks for this explanation.  It sounds like each filter actually has number of `input_channels` versions with __different weights__.  Do you have an "official" source that confirms this understanding?  
  > **username_5**: @RyanChase: Yes that is correct. I would just point you at Andrew Ng's course on CNNs - starting here with how colour image would be processed:  https://www.coursera.org/learn/convolutional-neural-networks/lecture/ctQZz/convolutions-over-volume  
  > **username_6**: I  would like to note that, in that source (http://cs231n.github.io/convolutional-networks/), filters (weights or kernesl) are volumes (i.e. 3-dimensional), and they have the same 3rd dimension has the one of the input volume. Furthermore, as it is (at least) now stated in that source, the volumes have been sliced across the 3rd dimension in order to visualize better the application of the filter to the input volume. I don't think that, in general, "there is a separate kernel defined for each input channel / output channel combination." is correct.  
  > **username_6**: Note that the filters (or kernels) are the weights that need to be learned (i.e. they are not fixed, but they are actually the parameters of the CNN). It might be that they are (i.e. the slices of the filter), at the end, the same across the 3rd dimension.  
  > **username_5**: @nbro: Yes you can *implement* a 2D convolution across multiple 2D slices as a single 3D convolution with the kernel depth same as number of channels. Mathematically this is identical to my description. You can also view it as a truncated fully-connected feed forward network with shared weights (many of which are zero). This answer focuses on what the view of 2D filters is, because the OP is asking about how the 2D filters are arranged. They may in fact be arranged into a larger 3D kernel, but they are still applied as 2D kernels using the "trick" that the 3D convolution is equivalent.  
  
  ---
  
  ### Answer by username_7 on 2018-11-25. Score: 8
  
  I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively:
  
  Let the example be as follows (wrt to 1 convolutional layer):
  
  * the input tensor is 9x9x5, i.e. 5 input channels, so `input_channels=5`
  * the filter/kernel size is 4x4 and the stride is 1
  * the output tensor is 6x6x56, i.e. 56 output channels, so `output_channels=56`
  * the padding type is 'VALID' (i.e. no padding)
  
  We note that:
  
  * since the input has 5 channels, the filter dimension becomes 4x4x5, i.e. there are 5 separate, unique 2D filters of size 4x4 (i.e. each has 16 weights); in order to convolve over the input of size 9x9x5 the filter becomes 3D and must be of size 4x4x5
  * therefore: for each input channel, there exists a distinct 2D filter with 16 different weights each. In other words, the number of 2D filters matches the number of input channels
  * since there are 56 output channels, there must be 56 3-dimensional filters W0, W1, ..., W55 of size 4x4x5 (cf. in the CS231 graphic there are 2 3-dimensional filters W0, W1 to account for the 2 output channels), where the 3rd dimension of size 5 represents the link to the 5 input channels (cf. in the CS231 graphic each 3D filter W0, W1 has the 3rd dimension 3, which matches the 3 input channels)
  * therefore: the number of 3D filters equals the number of output channels
  
  That convolutional layer thus contains:
  
  56 3-dimensional filters of size 4x4x5 (= 80 different weights each) to account for the 56 output channels where each has a value for the 3rd dimension of 5 to match the 5 input channels. In total there are
  
  `number_of_filters=input_channel*output_channels=5*56=280`
  
  2D filters of size 4x4 (i.e. 280x16 different weights in total).
  
  
  ---
  
  ### Answer by username_2 on 2018-11-26. Score: 3
  
  I recommend chapter 2.2.1 of my masters thesis as an answer. To add to the remaining answers:
  Keras is your friend to understand what happens:
  from keras.models import Sequential
  from keras.layers import Conv2D
  
  model = Sequential()
  model.add(Conv2D(32, input_shape=(28, 28, 3),
            kernel_size=(5, 5),
            padding='same',
            use_bias=False))
  model.add(Conv2D(17, (3, 3), padding='same', use_bias=False))
  model.add(Conv2D(13, (3, 3), padding='same', use_bias=False))
  model.add(Conv2D(7, (3, 3), padding='same', use_bias=False))
  model.compile(loss='categorical_crossentropy', optimizer='adam')
  
  print(model.summary())
  
  gives
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  conv2d_1 (Conv2D)            (None, 28, 28, 32)        2400      
  _________________________________________________________________
  conv2d_2 (Conv2D)            (None, 28, 28, 17)        4896      
  _________________________________________________________________
  conv2d_3 (Conv2D)            (None, 28, 28, 13)        1989      
  _________________________________________________________________
  conv2d_4 (Conv2D)            (None, 28, 28, 7)         819       
  =================================================================
  Total params: 10,104
  
  Try to formulate your options. What would that mean for the parameters if something else would be the case?
  Hint: $2400 = 32 \cdot (3 \cdot 5 \cdot 5)$
  This approach also helps you with other layer types, not only convolutional layers.
  Please also note that you are free to implement different solutions, that might have other numbers of parameters.
  
  
  ---
  
  ### Answer by username_8 on 2023-01-10. Score: 2
  
  One addition to Mohsin's answer: try the awesome [CNN explainer](https://poloclub.github.io/cnn-explainer/). There you can explore CNN's graphically, which makes things "click" very fast.
  
  If you click on conv\_1\_1, you will see how three different kernels are used to calculate one convolution. That means you have one 2D filter per channel, that could together be interpreted as one 3D filter.
  
  > **Comments:**
  > **username_9**: While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/23959)  
  
  ---
  
  ### Answer by username_6 on 2021-12-18. Score: 0
  
  Other answers claim that you have a different **2-dimensional** kernel (i.e. a matrix) for each channel. This is not wrong, but it's just a *conceptual interpretation* of the 2d convolution, which emphasizes that different channels may provide different information during the convolution: this is the only advantage that I see of this interpretation!
  
  Another interpretation (which I think has more advantages, as I will explain below) is to think of a kernel as a 3d multi-dimensional array that has the same depth as the input. So, for example, if you have an input (an image or feature map) of shape $H \times W \times D$, then a single kernel needs to have the shape $K \times L \times D$, where $D$ is the depth of both the input and kernel. If you have more than one kernel, they will all have the depth $D$. This interpretation is consistent with the excerpt that you are quoting.
  
  This second interpretation is conceptually useful when you need to deal with 3d convolutions: if you follow this interpretation, a 3d convolution is just a convolution where kernels don't necessarily have the same depth as the input (and that would be the only difference between 2d and 3d convolutions). The other advantage of this interpretation is that your confusion would not have arisen in the first place: your confusion arouse because you thought that kernels are 2d arrays. Another advantage is that, if you have $N > 1$ kernels (which is the usual case), then you know immediately that $N$ is the depth of the output volume and you don't need to think about multiple channels of the input volume: you can simply think in terms of multi-dimensional arrays.
  
  So, to clarify, in case it wasn't already super clear: **no, you don't apply the same 2d matrix to multiple channels of the input!**
  
  
  ---
  
  ### Answer by username_10 on 2019-02-26. Score: 0
  
  Just to make two details absolutely clear:
  
  Say you have $N$ 2D input channels going to $N$ 2D output channels. The total number of 2D $3\times3$ filter weights is actually $N^2$. But how is the 3D convolution affected, i.e., if every input channel contributes one 2D layer to every output channel, then each output channel is composed initially of $N$ 2D layers, how are they combined?
  
  This tends to be glossed over in almost every publication I've seen, but the key concept is the $N^2$ 2D output channels are interleaved with each other to form the $N$ output channels, like shuffled card decks, before being summed together. This is all logical when you realize that along the channel dimensions of a convolution (which is never illustrated), you actually have a fully connected layer! Every input 2D channel, multiplied by a unique $3\times 3$ filter, yields a 2D output layer contribution to a single output channel. Once combined, every output layer is a combination of every input layer $\times$ a unique filter. It's an all to all contribution.
  
  The easiest way to convince yourself of this is to imagine what happens in other scenarios and see that the computation becomes degenerate - that is, if you don't interleave and recombine the results, then the different outputs wouldn't actually do anything - they'd have the same effect as a single output with combined weights.
  
  
  ---
  
  ### Answer by username_11 on 2019-07-10. Score: 0
  
  For anyone trying to understand how convolutions are calculated, here is a useful code snippet in Pytorch:
  batch_size = 1
  height = 3 
  width = 3
  conv1_in_channels = 2
  conv1_out_channels = 2
  conv2_out_channels = 2
  kernel_size = 2
  # (N, C_in, H, W) is shape of all tensors. (batch_size, channels, height, width)
  input = torch.Tensor(np.arange(0, batch_size*height*width*in_channels).reshape(batch_size, in_channels, height, width))
  conv1 = nn.Conv2d(in_channels, conv1_out_channels, kernel_size, bias=False) # no bias to make calculations easier
  # set the weights of the convolutions to make the convolutions easier to follow
  nn.init.constant_(conv1.weight[0][0], 0.25)
  nn.init.constant_(conv1.weight[0][1], 0.5)
  nn.init.constant_(conv1.weight[1][0], 1) 
  nn.init.constant_(conv1.weight[1][1], 2) 
  out1 = conv1(input) # compute the convolution
  
  conv2 = nn.Conv2d(conv1_out_channels, conv2_out_channels, kernel_size, bias=False)
  nn.init.constant_(conv2.weight[0][0], 0.25)
  nn.init.constant_(conv2.weight[0][1], 0.5)
  nn.init.constant_(conv2.weight[1][0], 1) 
  nn.init.constant_(conv2.weight[1][1], 2) 
  out2 = conv2(out1) # compute the convolution
  
  for tensor, name in zip([input, conv1.weight, out1, conv2.weight, out2], ['input', 'conv1', 'out1', 'conv2', 'out2']):
      print('{}: {}'.format(name, tensor))
      print('{} shape: {}'.format(name, tensor.shape))
  
  Running this gives the following output:
  input: tensor([[[[ 0.,  1.,  2.],
            [ 3.,  4.,  5.],
            [ 6.,  7.,  8.]],
  
           [[ 9., 10., 11.],
            [12., 13., 14.],
            [15., 16., 17.]]]])
  input shape: torch.Size([1, 2, 3, 3])
  conv1: Parameter containing:
  tensor([[[[0.2500, 0.2500],
            [0.2500, 0.2500]],
  
           [[0.5000, 0.5000],
            [0.5000, 0.5000]]],
  
  
          [[[1.0000, 1.0000],
            [1.0000, 1.0000]],
  
           [[2.0000, 2.0000],
            [2.0000, 2.0000]]]], requires_grad=True)
  conv1 shape: torch.Size([2, 2, 2, 2])
  out1: tensor([[[[ 24.,  27.],
            [ 33.,  36.]],
  
           [[ 96., 108.],
            [132., 144.]]]], grad_fn=)
  out1 shape: torch.Size([1, 2, 2, 2])
  conv2: Parameter containing:
  tensor([[[[0.2500, 0.2500],
            [0.2500, 0.2500]],
  
           [[0.5000, 0.5000],
            [0.5000, 0.5000]]],
  
  
          [[[1.0000, 1.0000],
            [1.0000, 1.0000]],
  
           [[2.0000, 2.0000],
            [2.0000, 2.0000]]]], requires_grad=True)
  conv2 shape: torch.Size([2, 2, 2, 2])
  out2: tensor([[[[ 270.]],
  
           [[1080.]]]], grad_fn=)
  out2 shape: torch.Size([1, 2, 1, 1])
  
  Notice how the each channel of the convolution sums over all previous channels outputs.
  
  
  ---
  
  ### Answer by username_12 on 2021-11-22. Score: 0
  
  My understanding from this paper <https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/TASLP2339736-proof.pdf> was that the filters are used on each input channels (i.e input feature map in the paper) separately and the result is summed, as described in eq. 8. Here they use different filters for each channel, but you could totally use the same filter.
  As to knowing the software implementation in a particular ML library such as Tensorflow or PyTorch, this requires inspection as suggested by @Mohsin Bukhari
  
  
  ---
  
  ### Answer by username_13 on 2023-04-24. Score: 0
  
  The Conv2d filter is defined as a 3D-represented tensor, but it's indeed a collection of num\_input\_channels kernels applied laterally. The different kernels' layers' values are distinct and learned distinctly too. Theoretically it's necessary for it to be that way; otherwise it wouldn't have the degrees of freedom to distinguish well across the input channels, e.g. find color features in the case of the first layer of a color image input!
  
  
  ---
  
  ### Answer by username_14 on 2018-07-07. Score: 0
  
  Refer to "Local Connectivity" section in [here](http://cs231n.github.io/convolutional-networks/) and slide 7-18.
  
  "Receptive Field" hyperparameter of filter is defined by height & width only, as depth is fixed by preceding layer's depth.
  
  NOTE that "The extent of the connectivity along the depth axis is always equal to the DEPTH of the input volume" -or- DEPTH of activation map (in case of later layers).
  
  Intuitively, this must be due to the fact that image channels data are interleaved, not planar. This way, applying filter can be achieved simply by column vectors multiplication.
  
  NOTE that Convolutional Network learns all the filter parameters (including the depth dimension) and they are total "h*w*input\_layer\_depth + 1 (bias)".
  
  
  ---
  
  ### Answer by username_15 on 2018-03-23. Score: -1
  
  There are only restriction in 2D. Why?
  
  Imagine a fully connected layer.
  
  It'd be awfully huge, each neuron would be connected to maybe 1000x1000x3 inputs neurons. But we know that processing nearby pixel makes sense, therefore we limit ourselves to a small 2D-neighborhood, so each neuron is connected to only a 3x3 near neurons in 2D. We don't know such a thing about channels, so we connect to all channels.
  
  Still, there would be too many weights. But because of the translation invariance, a filter working well in one area is most probably useful in a different area. So we use the same set of weights across 2D. Again, there's no such translation invariance between channels, so there's no such restriction there.
  
  
  ---
  
  