Answers.Body: <p>You are partially correct.  On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.).  </p>
  
  <h2>CNNs shapes</h2>
  
  <p>In your example, your input is  <code>30 x 30 x 3</code>.  Assuming stride of <code>1</code>, no padding, and no dilation on the filter, you will get a spatial shape equal to your input, that is <code>30 x 30</code>.  Regarding the depth if you have <code>10</code> filters (of shape <code>5 x 5 x 3</code>) you will end up with a <code>30 x 30 x 10</code> output at your first layer.  Similarly, on the second layer with 5 filters (of shape <code>3 x 3 x 10</code>, note the depth to work on the previous layer) you have <code>30 x 30 x 5</code> output. The FC layer has the same amount of weights as the input (that is <code>4500</code> weights) to create a linear combination of them.</p>
  
  <h2>CNN vs Convolution</h2>
  
  <p>Note that the CNNs operate differently from the traditional signal processing convolution.  In the former, the convolution operation performs a dot product with the filter and the input to output a single value (and even add bias if you want to).  While the latter outputs the same amount of channels.</p>
  
  <p>The CNNs borrow the idea of a shifting kernel and a kernel response.  But they do not apply a convolution operation per se.  </p>
  
  <h2>Operation over the RGB</h2>
  
  <p>The CNN is not operating on each channel separately.  It is merging the responses of the three channels and mixing them further.  The deeper you get the more mix you get over your previous results.  </p>
  
  <p>The output of your FC is just one value.  If you want more, you need to add more FC neurons to get more linear combinations of your inputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T12:41:44.793Z
Answers.LastActivityDate: 2019-12-10T12:41:44.793Z
Answers.UserDisplayName: adn
Answers.Body: <p>About the images inside the CNN layers: I really recommend this <a href="https://becominghuman.ai/what-exactly-does-cnn-see-4d436d8e6e52" rel="nofollow noreferrer">article</a> since there is no one short answer to this question and it probably will be better to experiment with it.</p>
  
  <p>About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears, green body and blue tail. Each separated channel don't represent a cat, most certainly not with high confidence. I would recommend to transform you RGB images to gray scale and measure the network performance. If the performance are not sufficient you can make a 3D convolution layer. For example: If 30x30x3 is the input image, the filter has to be NxNx3.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T23:39:01.133Z
Answers.Text: Thanks for the link  I understand what you mean about losing information by processing the channels separately (independently). But I'm having difficulty finding a universal way to combine the them into one channel. Converting to grayscale (or "luminosity" using YCbCr conversion) may work in some cases, but not if for example there is a green gat on a red background. However a 3D convolution seems like overkill? Or is it reasonable to ignore these exceptions?
Answers.UserDisplayName: RocketNuts
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T22:21:19.243Z
Answers.LastActivityDate: 2019-12-09T22:21:19.243Z
Answers.UserDisplayName: Sahar Sela
Answers.Body: <p>For a 3 channel image (RGB), each filter in a convolutional layer computes a <strong>feature map</strong> which is essentially a single channel image.  Typically, 2D convolutional filters are used for multichannel images. This can be a single filter applied to each layer or a seperate filter per layer. These filters are looking for features which are independent of the color, i.e. edges (if you are looking for color there are far easier ways than CNNs). The filter is applied to each channel and the results are combined into a single output, the feature map. Since all channels are used by the filter to compute a single feature map, the number of channels in the input does not affect the structure of the network beyond the first layer. The size of a feature map is determined by the <strong>filter size</strong>, <strong>stride</strong>, <strong>padding</strong> and <strong>dilation</strong>(not commonly used - see <a href="https://towardsdatascience.com/understanding-2d-dilated-convolution-operation-with-examples-in-numpy-and-tensorflow-with-d376b3972b25" rel="nofollow noreferrer">here</a> if you are interested.).</p>
  
  <p>In your example, a <code>30 x 30 x 3</code> input convolved with 10 <code>5 x 5</code> filters will yield a volume of <code>30 x 30 x 10</code> <strong>if</strong> the filters have a stride of <strong>1</strong> and <strong>same</strong> padding (or, <code>26 x 26 x 10</code> with <strong>valid</strong> padding / <code>34 x 34 x 10</code> with <strong>full</strong> padding).</p>
  
  <p><strong>Same</strong> padding buffers the edge of the input with <em>filter_size</em>/2 (integer division) to yield an output of equal size (assuming stride is 1) while <strong>valid</strong> padding would result in a smaller output. Valid padding doesn't crop the image as you said, it's more of a dilution of the signal at the edges, however the results is essentially the same. Note that even with same padding the edge pixels are used in fewer convolutions - a <code>5 x 5</code> filter with same padding will use a central pixel 25 times (every position on the filter) but only 9 times for a corner pixel. To use all pixels evenly <strong>full</strong> padding must be used which buffers the edge of the input with <em>filter_size</em> - 1.</p>
  
  <p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href="https://i.sstatic.net/7HSXgm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7HSXgm.png" alt="enter image description here"></a></p>
  
  <p>Each feature map becomes a channel in the output volume. Therefore, <strong>the number of channels in the output volume is always equal to the number of filters in the convolutional layer</strong>. So, the second layer would output a volume of size <code>30 x 30 x 5</code> (stride 1, same padding).</p>
  
  <p>The last layer in your example (fully connected) multiplies the value of each pixel in each feature map by a learned weight and sums the result. If the network is a binary classifier, the summed value results in a 1 or 0 output if a threshold is reached or as a decimal value for a regression model. This is determined by the FC neurons' activation function.</p>
  
  <p>If visualizing this helps you as much as it helps me, I highly recommend having a look at the interactive examples <a href="https://tensorspace.org/html/playground/lenet.html" rel="nofollow noreferrer">here</a>. Note that what is shown by this tool is the signal propagating through the network, i.e. the output from each layer, not the filters/weights themselves. </p>
  
  <p>If you are interested in a bit more depth about ANNs and convolutional layers, I cover all the basics in my <a href="https://github.com/tynowell/rotdetection-CNN/blob/master/Detection%20and%20Quantification%20of%20Rot%20in%20Harvested%20Trees%20using%20Convolutional%20Neural%20Networks.pdf" rel="nofollow noreferrer">thesis</a>(this is where the image is from) - p.9-16 ANNs &amp; p.16-23 CNNs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T10:17:52.360Z
Answers.LastActivityDate: 2019-12-12T10:25:11.933Z
Answers.LastEditDate: 2019-12-12T10:25:11.933Z
Answers.UserDisplayName: tynowell
Body: <p>Say I have a CNN with this structure:</p>
  
  <ul>
  <li>input = 1 image (say, 30x30 RGB pixels)</li>
  <li>first convolution layer = 10 5x5 convolution filters</li>
  <li>second convolution layer = 5 3x3 convolution filters</li>
  <li>one dense layer with 1 output</li>
  </ul>
  
  <p>So a graph of the network will look like this:</p>
  
  <p><img src="https://i.sstatic.net/6aXFM.png" alt="Convolutional Neural Network"></p>
  
  <p>Am I correct in thinking that the first convolution layer will create 10 new images, i.e. each filter creates a new intermediary 30x30 image (or 26x26 if I crop the border pixels that cannot be fully convoluted).</p>
  
  <p>Then the second convolution layer, is that supposed to apply the 5 filters <strong>on all of the 10 images from the previous layer</strong>? So that would result in a total of 50 images after the second convolution layer.</p>
  
  <p>And then finally the last FC layer will take all data from these 50 images and somehow combine it into one output value (e.g. the probability that the original input image was a cat).</p>
  
  <p>Or am I mistaken in how convolution layers are supposed to operate? </p>
  
  <p>Also, how to deal with channels, in this case RGB? Can I consider this entire operation to be separate for all red, green and blue data? I.e. for one full RGB image, I essentially run the entire network three times, once for each color channel? Which would mean I'm also getting 3 output values.</p>
  
Tags: <convolutional-neural-networks><image-processing><convolution><hidden-layers><convolution-arithmetic>
Title: Does each filter in each convolution layer create a new image?
UserDisplayName: RocketNuts
ThreadText: # Does each filter in each convolution layer create a new image?
  
  (Asked by: username_0 on 2019-12-09)
  
  Say I have a CNN with this structure:
  
  * input = 1 image (say, 30x30 RGB pixels)
  * first convolution layer = 10 5x5 convolution filters
  * second convolution layer = 5 3x3 convolution filters
  * one dense layer with 1 output
  
  So a graph of the network will look like this:
  
  ![Convolutional Neural Network](https://i.sstatic.net/6aXFM.png)
  
  Am I correct in thinking that the first convolution layer will create 10 new images, i.e. each filter creates a new intermediary 30x30 image (or 26x26 if I crop the border pixels that cannot be fully convoluted).
  
  Then the second convolution layer, is that supposed to apply the 5 filters **on all of the 10 images from the previous layer**? So that would result in a total of 50 images after the second convolution layer.
  
  And then finally the last FC layer will take all data from these 50 images and somehow combine it into one output value (e.g. the probability that the original input image was a cat).
  
  Or am I mistaken in how convolution layers are supposed to operate?
  
  Also, how to deal with channels, in this case RGB? Can I consider this entire operation to be separate for all red, green and blue data? I.e. for one full RGB image, I essentially run the entire network three times, once for each color channel? Which would mean I'm also getting 3 output values.
  
  **Tags:** convolutional-neural-networks, image-processing, convolution, hidden-layers, convolution-arithmetic
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-12-10. Score: 4 | Accepted answer
  
  You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.).
  
  ## CNNs shapes
  
  In your example, your input is `30 x 30 x 3`. Assuming stride of `1`, no padding, and no dilation on the filter, you will get a spatial shape equal to your input, that is `30 x 30`. Regarding the depth if you have `10` filters (of shape `5 x 5 x 3`) you will end up with a `30 x 30 x 10` output at your first layer. Similarly, on the second layer with 5 filters (of shape `3 x 3 x 10`, note the depth to work on the previous layer) you have `30 x 30 x 5` output. The FC layer has the same amount of weights as the input (that is `4500` weights) to create a linear combination of them.
  
  ## CNN vs Convolution
  
  Note that the CNNs operate differently from the traditional signal processing convolution. In the former, the convolution operation performs a dot product with the filter and the input to output a single value (and even add bias if you want to). While the latter outputs the same amount of channels.
  
  The CNNs borrow the idea of a shifting kernel and a kernel response. But they do not apply a convolution operation per se.
  
  ## Operation over the RGB
  
  The CNN is not operating on each channel separately. It is merging the responses of the three channels and mixing them further. The deeper you get the more mix you get over your previous results.
  
  The output of your FC is just one value. If you want more, you need to add more FC neurons to get more linear combinations of your inputs.
  
  
  ---
  
  ### Answer by username_2 on 2019-12-09. Score: 4
  
  About the images inside the CNN layers: I really recommend this [article](https://becominghuman.ai/what-exactly-does-cnn-see-4d436d8e6e52) since there is no one short answer to this question and it probably will be better to experiment with it.
  
  About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears, green body and blue tail. Each separated channel don't represent a cat, most certainly not with high confidence. I would recommend to transform you RGB images to gray scale and measure the network performance. If the performance are not sufficient you can make a 3D convolution layer. For example: If 30x30x3 is the input image, the filter has to be NxNx3.
  
  > **Comments:**
  > **username_0**: Thanks for the link  I understand what you mean about losing information by processing the channels separately (independently). But I'm having difficulty finding a universal way to combine the them into one channel. Converting to grayscale (or "luminosity" using YCbCr conversion) may work in some cases, but not if for example there is a green gat on a red background. However a 3D convolution seems like overkill? Or is it reasonable to ignore these exceptions?  
  
  ---
  
  ### Answer by username_3 on 2019-12-12. Score: 4
  
  For a 3 channel image (RGB), each filter in a convolutional layer computes a **feature map** which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel images. This can be a single filter applied to each layer or a seperate filter per layer. These filters are looking for features which are independent of the color, i.e. edges (if you are looking for color there are far easier ways than CNNs). The filter is applied to each channel and the results are combined into a single output, the feature map. Since all channels are used by the filter to compute a single feature map, the number of channels in the input does not affect the structure of the network beyond the first layer. The size of a feature map is determined by the **filter size**, **stride**, **padding** and **dilation**(not commonly used - see [here](https://towardsdatascience.com/understanding-2d-dilated-convolution-operation-with-examples-in-numpy-and-tensorflow-with-d376b3972b25) if you are interested.).
  
  In your example, a `30 x 30 x 3` input convolved with 10 `5 x 5` filters will yield a volume of `30 x 30 x 10` **if** the filters have a stride of **1** and **same** padding (or, `26 x 26 x 10` with **valid** padding / `34 x 34 x 10` with **full** padding).
  
  **Same** padding buffers the edge of the input with *filter\_size*/2 (integer division) to yield an output of equal size (assuming stride is 1) while **valid** padding would result in a smaller output. Valid padding doesn't crop the image as you said, it's more of a dilution of the signal at the edges, however the results is essentially the same. Note that even with same padding the edge pixels are used in fewer convolutions - a `5 x 5` filter with same padding will use a central pixel 25 times (every position on the filter) but only 9 times for a corner pixel. To use all pixels evenly **full** padding must be used which buffers the edge of the input with *filter\_size* - 1.
  
  [![enter image description here](https://i.sstatic.net/7HSXgm.png)](https://i.sstatic.net/7HSXgm.png)
  
  Each feature map becomes a channel in the output volume. Therefore, **the number of channels in the output volume is always equal to the number of filters in the convolutional layer**. So, the second layer would output a volume of size `30 x 30 x 5` (stride 1, same padding).
  
  The last layer in your example (fully connected) multiplies the value of each pixel in each feature map by a learned weight and sums the result. If the network is a binary classifier, the summed value results in a 1 or 0 output if a threshold is reached or as a decimal value for a regression model. This is determined by the FC neurons' activation function.
  
  If visualizing this helps you as much as it helps me, I highly recommend having a look at the interactive examples [here](https://tensorspace.org/html/playground/lenet.html). Note that what is shown by this tool is the signal propagating through the network, i.e. the output from each layer, not the filters/weights themselves.
  
  If you are interested in a bit more depth about ANNs and convolutional layers, I cover all the basics in my [thesis](https://github.com/tynowell/rotdetection-CNN/blob/master/Detection%20and%20Quantification%20of%20Rot%20in%20Harvested%20Trees%20using%20Convolutional%20Neural%20Networks.pdf)(this is where the image is from) - p.9-16 ANNs & p.16-23 CNNs.
  
  
  ---
  
  