Answers.Body: <p>I don't think that to understand convolution you need to dig into the nested code of huge libraries, since the code becomes quickly really hard to understand and convoluted (ba dum tsss!). Joking apart, in PyTorch <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d" rel="nofollow noreferrer">Conv2d</a> is a layer that applies another low level function, <a href="https://github.com/pytorch/pytorch/blob/c780610f2d8358297cb4e4460692d496e124d64d/aten/src/ATen/native/Convolution.cpp#L481" rel="nofollow noreferrer">conv2d</a>, written in c++. </p>
  
  <p>Luckily enough, the guys from PyTorch wrote the general idea of how convolution is implemented in the documentation:</p>
  
  <blockquote>
    <p><a href="https://i.sstatic.net/ToMaM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ToMaM.png" alt="enter image description here"></a></p>
  </blockquote>
  
  <p>From this paragraph, we already have some important information like the input and output dimensions. The number of channels should be easy to understand, if we have an RGB image, for example, the channels are 3, one for each color, so they are just different matrices representing different features. </p>
  
  <p>The next important element is the reference to the <a href="https://en.wikipedia.org/wiki/Cross-correlation" rel="nofollow noreferrer">cross correlation</a>, the function applied to our input images through the kernel k. Why cross-correlation? Because it is almost identical to a convolution, as you can see comparing their formulas:</p>
  
  <p><a href="https://i.sstatic.net/Bmns0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Bmns0.jpg" alt="enter image description here"></a></p>
  
  <p>The only difference consists in the way the indexing of the width is implemented, which causes the operation to start from the bottom right of the input matrix for the convolution and from the top left of the input image from the cross-correlation (the circles in the squares in the previous pic). Since, in most programming languages, matrix indexing starts from top left, cross-correlation is the most common choice to implement. </p>
  
  <p>But how do these formulas work in practice? Here's another picture taken from Chapter 9 of <a href="http://www.deeplearningbook.org/contents/convnets.html" rel="nofollow noreferrer">Deep Learning</a> (Goodfellow, Bengio, Courville), which I strongly suggest you to read.</p>
  
  <p><a href="https://i.sstatic.net/wZsdt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wZsdt.png" alt="enter image description here"></a></p>
  
  <p>Basically, from the input matrix, a submatrix is extracted, with the same dimension of the kernel, then the sub-matrix and the kernel are multiplied <strong>elementwise</strong> and all the resulting product summed together to produce a single output element that will form a 'pixel' of the resulting <em>feature map</em> (the output matrix).</p>
  
  <p>Here's another example with fake numbers that I made. I hope the double notation for filter/kernel doesn't generate confusion, I actually found that sometimes it is inconsistent (in the chapter I linked they don't even use the filter at all). In practice they actually mean the same thing, I usually call kernel the actual matrix that is multiplied to the input and with filter I refer to the sliding window on the input image (that has, of course, must have the same dimension of the kernel). </p>
  
  <p><a href="https://i.sstatic.net/0DmF4.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0DmF4.jpg" alt="enter image description here"></a></p>
  
  <p>Lastly, when you apply padding, the filter can actually move also outside the 'edges' of the input matrix, in which case all the elements outside would be considered to be zero. The computation is exactly the same, but since there are more splitting steps, the output matrix will have bigger dimension.</p>
  
  <p><a href="https://i.sstatic.net/nTXy3.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nTXy3.jpg" alt="enter image description here"></a></p>
  
  <p>Please note that with multiple input channels you can perform either 2d convolution or 3d convolution, the difference relies in the filter dimension, in case of 2d convolution it would be a square whereas in 3d convolution it would be a cube. This means that for an RGB image a 2d convolution would treat each color layer independently, mixing the information from each channel with further computations like pooling (averaging the resulting feature maps of each color or select the max value among the feature maps for each pixel, etc..) while 3d convolution would mix together the color layers already during convolution thanks to the 3d kernel which will sum together elements from different layers. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-15T22:06:06.353Z
Answers.LastActivityDate: 2020-04-15T22:29:19.307Z
Answers.LastEditDate: 2020-04-15T22:29:19.307Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Following an earlier question, I'm interested in understanding the basics of <code>Conv2d</code> and especially how the kernel is applied, summed, and the propagated.
  I understand that:</p>
  
  <ol>
  <li>a <code>kernel</code> has size <code>W</code> x <code>H</code> and that more than one kernel is applied (e.g., S x W x H) where <code>S</code> is the amount of kernels.</li>
  <li>A stride or step is used when iterating the network input</li>
  <li>Padding may or may not be used by being added to the network input.</li>
  </ol>
  
  <p>What I would ideally like to see is either a description, or a python sample (<code>pytorch</code> or <code>tensorflow</code>) of how that is done, what the dimensionality of the output is, and any operation I may be missing (some YouTube videos say that the kernel is summarised and then divided to hold one new unique value representing the feature activation?)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T16:18:33.357Z
Comments.Text: It's not clear to me if you're asking for a general description of how the convolution layer works or just some particular detail, like how to calculate the shape of the output of a convolution layer.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T16:20:19.930Z
Comments.Text: @nbro I'd say I'm asking for the implementation, either in a math level, or a pseudo code level, or even better as a minified example? The basics of Conv2D and how that works on the input and how the output is produced. I understand there are other parts and many variants.
Comments.UserDisplayName: Ælex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T16:22:01.230Z
Comments.Text: Well, the implementation may be different than the actual general idea. So, I suggest you edit your post to clarify that you're specifically looking for "how the convolution layer is usually implemented in practice"
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T16:35:51.013Z
Comments.Text: @nbro done :-) Only reason I'm asking for the implementation is because it tends to be more easily digestible than the theory more often than not.
Comments.UserDisplayName: Ælex
Tags: <convolutional-neural-networks><implementation><convolution>
Title: How is the convolution layer is usually implemented in practice?
UserDisplayName: Ælex
ThreadText: # How is the convolution layer is usually implemented in practice?
  
  (Asked by: username_0 on 2020-04-15)
  
  Following an earlier question, I'm interested in understanding the basics of `Conv2d` and especially how the kernel is applied, summed, and the propagated.
  I understand that:
  
  1. a `kernel` has size `W` x `H` and that more than one kernel is applied (e.g., S x W x H) where `S` is the amount of kernels.
  2. A stride or step is used when iterating the network input
  3. Padding may or may not be used by being added to the network input.
  
  What I would ideally like to see is either a description, or a python sample (`pytorch` or `tensorflow`) of how that is done, what the dimensionality of the output is, and any operation I may be missing (some YouTube videos say that the kernel is summarised and then divided to hold one new unique value representing the feature activation?)
  
  **Tags:** convolutional-neural-networks, implementation, convolution
  
  > **Comments:**
  > **username_1**: It's not clear to me if you're asking for a general description of how the convolution layer works or just some particular detail, like how to calculate the shape of the output of a convolution layer.  
  > **username_0**: @username_1 I'd say I'm asking for the implementation, either in a math level, or a pseudo code level, or even better as a minified example? The basics of Conv2D and how that works on the input and how the output is produced. I understand there are other parts and many variants.  
  > **username_1**: Well, the implementation may be different than the actual general idea. So, I suggest you edit your post to clarify that you're specifically looking for "how the convolution layer is usually implemented in practice"  
  > **username_0**: @username_1 done :-) Only reason I'm asking for the implementation is because it tends to be more easily digestible than the theory more often than not.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-15. Score: 4 | Accepted answer
  
  I don't think that to understand convolution you need to dig into the nested code of huge libraries, since the code becomes quickly really hard to understand and convoluted (ba dum tsss!). Joking apart, in PyTorch [Conv2d](https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d) is a layer that applies another low level function, [conv2d](https://github.com/pytorch/pytorch/blob/c780610f2d8358297cb4e4460692d496e124d64d/aten/src/ATen/native/Convolution.cpp#L481), written in c++.
  
  Luckily enough, the guys from PyTorch wrote the general idea of how convolution is implemented in the documentation:
  
  > [![enter image description here](https://i.sstatic.net/ToMaM.png)](https://i.sstatic.net/ToMaM.png)
  
  From this paragraph, we already have some important information like the input and output dimensions. The number of channels should be easy to understand, if we have an RGB image, for example, the channels are 3, one for each color, so they are just different matrices representing different features.
  
  The next important element is the reference to the [cross correlation](https://en.wikipedia.org/wiki/Cross-correlation), the function applied to our input images through the kernel k. Why cross-correlation? Because it is almost identical to a convolution, as you can see comparing their formulas:
  
  [![enter image description here](https://i.sstatic.net/Bmns0.jpg)](https://i.sstatic.net/Bmns0.jpg)
  
  The only difference consists in the way the indexing of the width is implemented, which causes the operation to start from the bottom right of the input matrix for the convolution and from the top left of the input image from the cross-correlation (the circles in the squares in the previous pic). Since, in most programming languages, matrix indexing starts from top left, cross-correlation is the most common choice to implement.
  
  But how do these formulas work in practice? Here's another picture taken from Chapter 9 of [Deep Learning](http://www.deeplearningbook.org/contents/convnets.html) (Goodfellow, Bengio, Courville), which I strongly suggest you to read.
  
  [![enter image description here](https://i.sstatic.net/wZsdt.png)](https://i.sstatic.net/wZsdt.png)
  
  Basically, from the input matrix, a submatrix is extracted, with the same dimension of the kernel, then the sub-matrix and the kernel are multiplied **elementwise** and all the resulting product summed together to produce a single output element that will form a 'pixel' of the resulting *feature map* (the output matrix).
  
  Here's another example with fake numbers that I made. I hope the double notation for filter/kernel doesn't generate confusion, I actually found that sometimes it is inconsistent (in the chapter I linked they don't even use the filter at all). In practice they actually mean the same thing, I usually call kernel the actual matrix that is multiplied to the input and with filter I refer to the sliding window on the input image (that has, of course, must have the same dimension of the kernel).
  
  [![enter image description here](https://i.sstatic.net/0DmF4.jpg)](https://i.sstatic.net/0DmF4.jpg)
  
  Lastly, when you apply padding, the filter can actually move also outside the 'edges' of the input matrix, in which case all the elements outside would be considered to be zero. The computation is exactly the same, but since there are more splitting steps, the output matrix will have bigger dimension.
  
  [![enter image description here](https://i.sstatic.net/nTXy3.jpg)](https://i.sstatic.net/nTXy3.jpg)
  
  Please note that with multiple input channels you can perform either 2d convolution or 3d convolution, the difference relies in the filter dimension, in case of 2d convolution it would be a square whereas in 3d convolution it would be a cube. This means that for an RGB image a 2d convolution would treat each color layer independently, mixing the information from each channel with further computations like pooling (averaging the resulting feature maps of each color or select the max value among the feature maps for each pixel, etc..) while 3d convolution would mix together the color layers already during convolution thanks to the 3d kernel which will sum together elements from different layers.
  
  
  ---
  
  